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ABSTRACT 


The Network Centric Warfare approach to command and control emphasizes 
decentralized decision making. Consequently, decision makers must comprehend and 
evaluate information to determine the optimal course of action. This study examines how 
different categories of uncertainty (ambiguous/missing, conflicting, baseline) and 
individual differences affect response time in decision making tasks. The researchers 
elicited real-world tactical scenarios from veterans of Operation Enduring Freedom and 
Operation Iraqi Freedom in which uncertainty was present. Nine scenarios were given to 
28 participants at the Command General Staff College, FT Leavenworth, KS. The 
participants were asked to make a decision; their responses were recorded and analyzed. 
The results indicate that the category of uncertainty and scenario difficulty are significant 
factors in determining response time. No individual difference factors were found to be 
significant. These findings have the potential to improve human behavior modeling, 


tactical simulations, and representations of complex task environments. 
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EXECUTIVE SUMMARY 


The Information Age has revolutionized command and control in the tactical 
environment. The advent of Network Centric Warfare has led to an emphasis on 
decentralized decision making in real-time. The warfighter is expected to evaluate and 
analyze large quantities of information to produce a logical, rational decision at a 
moment’s notice. However, warfare often is neither rational nor logical. Although 
military commanders attempt to plan fail-safe operations, uncertainty often will create 
havoc or “fog” in the battle space. This fog forces commanders to adjust their plans in 
real time by using the information available to them. 

To apply their best judgment, commanders must utilize various inputs, including 
machine and human agents, to achieve an accurate awareness of the situation. This 
reliance on both human and machine agents strongly suggests a Human Systems 
Integration (HSI) approach - an approach that emphasizes the symbiotic, interdependent 
nature of human and machine agents to solve difficult and uncertain problems. 

During this study, U.S. Army officers, with recent experience in Operation 
Enduring Freedom and Operation Iraqi Freedom, were interviewed. The purpose of these 
interviews was to elicit real-world, tactical situations where uncertainty affected the 
decision making process. These scenarios were then crafted into nine vignettes 
representing the categories of uncertainty (i.e., ambiguous/missing, conflicting, and 
baseline). The vignettes were presented to 28 U.S. Army officers at FT Leavenworth, KS. 
The participants were asked to determine what decision they would make for each 
situation. Their decisions and response times were recorded and analyzed. Additionally, 
the participants completed a demographic survey and the Uncertainty Response Scale 
(e.g., emotional uncertainty & cognitive uncertainty). The researchers hypothesized that 
there would be a difference between the categories of uncertainty with respect to 
response time. Additionally, it was believed that emotional uncertainty, cognitive 


uncertainty, operational experience, and response time would be significantly correlated. 


XV 


The results suggest that the category of uncertainty coupled with scenario 
difficulty were significant factors in determining response time. Moreover, 
ambiguous/missing scenario response times were significantly greater than conflicting 
and baseline scenario response times. Operational experience, cognitive uncertainty, 
emotional uncertainty, and response time were not significantly correlated. 

In conclusion, the findings have implications for advancing the state of the art in 
modeling and simulation. This research provides insights into human behavior under 
uncertain conditions. These insights can aid researchers in more accurately modeling 
human behavior in tactical environments. The findings also extend to civilian and 


military decision making under complex tasking. 
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I. INTRODUCTION 


History is captured by moments of uncertainty. From the ancient and mythical 
battlefields of the Peloponnese to the rugged and unpredictable terrain of the War on 
Terror, statesmen, politicians, and military commanders have made critical decisions with 
uncertain information. This is the nature of the world we live in; very few things are 


certain, and most things are uncertain. 


This natural phenomenon has prompted many philosophers and theorists to 
address and pontificate on uncertainty’s role in the natural world. Consequently, 
psychologists, historians, and researchers have approached uncertainty from cognitive, 
historical, and sociologists’ perspectives. More importantly, they have attempted to 
define uncertainty, determine the effects, and discover its origins. Their research has 
yielded varying and numerous results. Regardless, their research has established a great 
foundation for future inquiry, and for this study, the effects of uncertainty on decision- 


making. 


A. THE HISTORICAL PREMISE OF UNCERTAINTY 


Commentary on uncertainty can be traced back to the greatest thinkers in Western 
literature and philosophy. Although the constructions of their arguments do not attempt 
to study uncertainty, they do make reference, and find its presence important in their 
works. In Discourse of Method, Rene Descartes (1637) addressed the importance of 
discovery and truth through science. To accomplish this, he asked man to “reject all 
previous knowledge, opinion, and customs” (Descartes, 1637, p. 21). He writes: 

The first was never to accept anything as true which I did not clearly know 

to be such, that is to say, carefully to avoid haste and prejudgment, and to 


accept nothing as true except what was presented to my mind so clearly 
and distinctly as to exclude all possibility of doubt. 


Descartes was arguing two points. First, knowledge, and therefore truth, cannot, 
and must not, be concluded without a thorough investigation of the question posed. 


Second, a statement cannot be considered true unless all uncertainty (1.e., doubt) is 


removed. The latter of the two is most important for this debate, because Descartes is 
advocating that doubt (i.e., uncertainty) naturally exists in any, and all, scientific and 


intellectual arguments. 


Moreover, Thomas Hobbes (1651) discussed this innate uncertainty through the 
nature of man. In The Leviathan, the Englishman examined the relationship between 
knowledge, opinion, judgment, and faith. Hobbes writes, “no discourse whatsoever can 
end in absolute knowledge of fact, past, or to come...no man can know by discourse, that 
this, or that, is, has been, or will be; which is to know absolutely” (p. 511). Therefore, the 
search for understanding can never conclude in concrete truth, because man cannot 
adequately predict the future or what is to come. More importantly, his judgment and 


reason can be clouded by personal conscious and faith. 


However, there is something more important to be learned from the writings of 
Descartes and Hobbes. The philosophers articulate the symbiosis of the natural state of 
man and the events of history. Although this may seem trivial, it supports the premise of 
uncertainty in the modern world; because man is naturally flawed, favored by opinion 
and prejudgment, unable to grasp all possible knowledge, and his affairs dynamic, 
uncertainty will always be present in all of man’s institutions. Thus, economics, politics, 
and for this study, warfare, by nature, will be laced with uncertainty. Consequently, it 


must be studied and understood. 


B. UNCERTAINTY IN WARFARE 


The role of uncertainty in warfare has been documented throughout history. 
Recently, Secretary of Defense Donald Rumsfeld, trying to explain the clouded future of 
the War on Terror, delivered a set of shrouded words that left the media confused and 
hysterical. In under a minute, the battle-hardened patriot described the innate 
characteristics of war. 

There are things that we know. There are known unknowns. That is to say, 

there are things that we know we don't know. But there are also unknown 


unknowns. There are things we don't know we don't know (Rumsfeld, 
2002). 


Although many scoffed, laughed, and viewed his uncanny remarks as mysterious, 
the proposed message was intended to prepare and warn the American public of the 
complex, dynamic nature of warfare. Like Hobbes’ interpretation of human nature, 


Rumsfeld viewed uncertainty as an inherent attribute of warfare. 


However, Rumsfeld’s often criticized, yet sometimes revered, speech was not the 
first to describe the assumptions, estimates, and uncertain nature of armed-conflict. More 
than two millennia earlier, military philosopher Sun Tzu (650) argued that the factors of 
war can be controlled; therefore uncertainty and chaos can be spawned, developed, and 
adjusted to confuse an opponent. Victory, therefore, can be achieved by creating a 


specific amount of uncertainty that both paralyzes and strikes fear in one’s adversary. 


Conversely, in the 19" century, military theorist Carl von Clauswitz (1805) 
argued that uncertainty is not a variable that can be manipulated in armed-conflict. 
Instead, he argued uncertainty to be a powerful and uncontrollable force that man is 
incapable of harnessing. Clauswitz described the climate of war using four distinct 
characteristics: danger, exertion, uncertainty, and chance. He wrote, “war is the realm of 
uncertainty; three quarters of the factors on which action in war is based are wrapped in a 
fog of greater or lesser uncertainty” (Clauswitz, 1805, p. 102). Consequently, uncertainty 
creates “friction” that makes the simplest movements difficult. To Clauswitz, war is not 
solely an act of pure intellect or intuition, but rather a delicate congruence of fate, luck, 


and skill. 


Interestingly, Sir Jillian Corbett (1911) may have best expressed the unique nature 
of man and uncertainty in war. A student of Clauswitz and Jomini, the British historian 
disagrees with the idea that war is an art vice a science with “no fixed laws or rules” 
(Corbett, 1911, p. 8). Instead, Corbett argues that history shows “certain lines of conduct 
produce certain effects” (p. 8). However, Corbett later suggests that “strategic analysis 
can never give exact results...only at approximations...” and will always “leave much to 


judge” (p. 83). 


His statements give more credence to Clauswitz’s views; that war is not an 
equation of set variables, but rather an open system of ever changing inputs and outputs. 
Therefore, uncertainty will always find a place among the commander’s sure plans or the 


infantryman’s charge. 


C. HUMAN SYSTEMS INTEGRATION 


To explore the role of uncertainty in military decision making, our study will 
address the topic from the perspective of Human Systems Integration (HSI). The 
definition of HSI continues to be the subject of debate. However, two definitions best 
describe the essence of the discipline. First, HSI is “the interdisciplinary approach that 
makes elicit the underlying tradeoffs across the domains, facilitating optimization of total 
system performance,” (Miller & Shattuck, 2006). Second, HSI is “a human centered 
approach to the design, development, and acquisition of systems from cradle to grave,” 
(Read, 2007, p. 3). In simpler terms, HSI aims to incorporate the human system in the 
“systems of systems” architecture; it attempts to remedy an individual’s weaknesses and 
highlight their strengths. In order to do so, HSI focuses on the eight domains and the 


tradeoffs within each. The following are the eight domains of HSI: 


e Manpower 

e Training 

e Personnel 

e Survivability 


e Habitability 

e Human Factors Engineering 

e Health Hazards 

e System Safety 

By incorporating these eight domains early in the acquisition cycle, the U.S. 
Department of Defense hopes to reduce costs, human survivability, and system lifespan. 
Further explanation of these terms can be found in Department of Defense Instruction 
5000.2 (U.S. Department of Defense, 2003) or Handbook of Human Systems Integration 
(Booher, 2003). 


For the scope of this study, the exploration of uncertainty in tactical decision 
making will be focused on modeling and simulation. Therefore, close attention will be 
paid to the domains of human factors engineering, personnel, and training. The 
researchers’ hope is to create algorithms for decision making under uncertainty in tactical 
simulations, thus providing a more realistic interpretation of the actual events of the 


battle space. 


D. RESEARCH QUESTIONS 


The historical presence of uncertainty and its influence on military history has 
prompted several important questions. While these questions provide an adequate 
foundation for research in the field of uncertainty, we must remind ourselves that the 
problem reaches far into the battle space. When addressing the problem from the 
perspective of HSI, the machine and human must be discussed in integrated, symbiotic 
terms. The following research questions were not mutually exclusive of the human being, 


but instead created to better understand the human’s cognitive abilities in combat. 


e What is uncertainty? What is its origin? What is its definition? 

e How does uncertainty affect the man and machine symbiosis? 

e What models and taxonomies can aid in the understanding of uncertainty 
in HSI? 

e What are the relationships between uncertainty, data, and individual 


difference factors? 


e Does an individual’s emotional or cognitive response to uncertainty affect 
his/her ability to cope with uncertainty? 


e Do decision makers delay their decisions when faced with uncertainty? 
E. RESEARCH OBJECTIVES 


With the premise properly stated and research questions posed, the objectives for 


the study can be outlined: 


e Create a methodology for examining uncertainty in the tactical 
environment. 
e Emphasize the importance of HSI in exploring uncertainty across the 


battle space. 


e Begin to identify the individual difference factors that influence decisions 
under uncertain conditions. 


e Develop of algorithms for modeling and simulation. 


e Provide a clear direction for future research in tactical decision making 
under uncertainty. 


F. THESIS ORGANIZATION 


The organization of this thesis follows a traditional format. In accordance with 
APA standards, it documents a literature review, methodology, results chapter, and 
discussion. Chapter II uses a top-down approach to examine decision-making under 
uncertainty through the perspective of HSI. Literature regarding command and control 
(C2), decision-making models, uncertainty, personality characteristics, and experience 


are analyzed and discussed to better explain their role in the realm of the uncertainty. 


Chapter III focuses on the previous pilot studies performed at NPS, along with the 
methodology used to conduct the final study. The pilot studies provide an interesting and 
important tale when addressing the nature of uncertainty. Additionally, “lessons learned” 
are documented and translated into future success. Moreover, a detailed description of the 


methodology is provided for future replication. 


Chapter IV and Chapter V provide statistical analysis and a discussion of the 
results. Additionally, conclusions based on previous research will be documented, along 


with opportunities for future research in the field of uncertainty. 


Il. LITERATURE REVIEW 


A. DECISION MAKING AND COMMAND AND CONTROL 
1. Network-Centric Warfare 


To remedy the effects of uncertainty, militaries have established means of 
communications and procedures to relay plans, monitor execution, and provide 
assessment (or assess results). These means of command and control (C2) remain central 
and increasingly important to all military operations and allow commanders to issue 


explicit and implicit orders when executing simple and complex operations. 


The theory of C2 has evolved over the course of military history. However, the 
Industrial Age forever altered its essence. The 20" century ushered in the use of machines 
in the battle space. Ships, aircraft, and tanks greatly increased the range, distance, 
lethality, and speed of battle. Additionally, radio and radar allowed commanders to view 
the enemy at great distances and communicate over thousands of miles of terrain and 
ocean. Consequently, platform-centric warfare became the preferred method of C2 
operations (i.e., the commander makes a centralized decision and communicates his 
direction to subordinate platforms who then execute the commanders’ orders) (Alberts, 
Garstka, & Stein, 1999; Alberts, 2003; Cebrowski & Garstka, 1998). More importantly, 
the warfighters were no longer autonomous in their own thought or action, but rather 
guided, and sometimes aided or instructed by mechanized technology (i.e., the integration 


of the human and machine system were vital to operational success). 


The evolution of technology (e.g., satellite communication, AEGIS, and 
simulation) soon gave new capabilities to commanders and operators. For the first time in 
warfare, the human possessed the capability to share, analyze, disseminate, and 
internalize distant battle space information in near real-time. Consequently, a new vision 
of military C2 operations emerged, and Cebrowski and Garstka’s (1998) Network Centric 
Warfare (NCW) theory changed the direction of C2 and military decision-making. 


The NCW concept has revolutionized military operations. The centralized 
information and decision-making environment of the past is being replaced by a shared 
network, focused on decentralized decision-making. Consequently, DeLange (2006) 
argues that NCW is merely a named process and the underlying, important message is 
“Decision Centric Warfare” (DCW) (i.e., improved decision-making is the most 
important aspect behind the NCW revolution). But while the particulars and details of 
NCW operations are fascinating and intriguing, the most important concepts lie in its four 
domains (cognitive, social, physical, and information) outlined by its creators (Cebrowski 
& Garstka, 1998). More importantly, for this study, how the cognitive domain impacts 
Network Centric Operations (NCO) when clouded by uncertainty 


2. The Cognitive Domain 


The cognitive domain is the mind of the decision maker (Alberts, Garstka, Hayes, 
& Signori, 2001). It serves as the most important element in NCO. While automation in 
complex systems has increased over the last quarter century, the decision maker 
continues to play the most important role in the system. Consequently, the ultimate 
success of NCW relies on the integration of human cognition and technological capability 


(Read, 2007, & Baker, 2002). 


To better understand the capabilities and limitations of human cognition in NCO, 
Garstka and Alberts (2004) have described the human’s thought process as 
“sensemaking,’ while also placing an emphasis on “shared sensemaking” or 
“collaboration.” The latter articulates two fundamental ideas: (1) the cognitive domain is 
influenced by the social domain (i.e., culture, organization, politics), and (2) the decision 
makers share their evaluation of the situation with their peers (Garstka & Alberts, 2004). 
Consequently, an individual decision maker can have a major impact, through a pseudo- 
social medium, on the course of action at both the micro and macro level due to the 
ability to share and analyze information in real time. Additionally, Caneva (1999) 
suggests NCW has the potential to not only increase military operations spatially, but also 
temporally. Therefore, the cognitive domain can expand the battlefield beyond our own 


expectations and imagination. 


While these points are important to recognize, it is outside the scope of our 
research. Instead, our goal is to focus on the “sensemaking” occurring at the individual 
cognitive level prior to peer-sharing. To describe the aforementioned, Garstka and 
Alberts (2004) explain the development of mental models to facilitate decision-making. 
Additionally, the researchers argue, “over time and with experience, people build up a 
repertoire of mental models that apply across a range of situations” (Garstka & Alberts, 
2004, p. 30). Their researcher implicitly hints at the various thought processes applied to 


decision-making. 


B. DECISION-MAKING MODELS 


To describe how individuals make or should make decisions, researchers have 
developed three models: (1) normative, (2) descriptive, and (3) naturalistic. Although all 
three models describe the human thought process in unique and explicit terms, the 
naturalistic model, or naturalistic decision-making model (NDM), has been widely 
accepted in recent years due to its utility in operational settings (Shattuck, 2007). More 
importantly, the NDM characteristics outlined by Orasanu and Connolly (1993) (e.g., ill- 
structured problems, uncertain dynamic environments, multiple players, and 
organizational norms and goals) describe the situations and context of military exercises 
and engagements (e.g., the recent 2008 military showdown between the United States and 


Iranian naval forces in the Straits of Hormuz). 


1. Naturalistic Decision Making 


In examining the utility of NDM in operational settings, Zsambok and Klein’s 
(1997), Naturalistic Decision Making, cites several real-world, research studies. For 
example, Hutchins (1997) addresses human error and situation awareness in U.S. Navy 
Littoral Anti Air Warfare (AAW) operations, while Serfaty, MacMillan, Entin, and Entin 


(1997) examined the importance of expertise in military tactical decision-making. 


Coincidentally, the AEGIS training center’s Tactical Decision Making under 
Stress (TADMUS) program focused on decision-making in stressful and time-sensitive 


environments (Cannon-Bowers & Salas, 1998). Additionally, Kaempf and Klein (1993) 


examined the decision-making strategies of naval officers aboard U.S. Navy AEGIS 
Cruisers. The aggregate research illustrates the complex, uncertain environments in 


which decision makers must operate. 


Consequently, NDM researchers have developed overlapping theories on how 
decision makers formulate and select their courses of action. For example, Klein’s (1993) 
Recognition Primed Decision-Making (RPD) model argues that decision makers rely on 
past experiences to select their courses of action (i.e., individuals do not select from 
several different outcomes, but rather choose their course of action based on past 
experience) (see Figure 1). He describes the model as consisting of two fused parts: (1) 
situational assessment and (2) mental simulation (Klein, 1993). Klein argues mental 
situational assessment generates a plausible course of action (COA) and mental 
simulation evaluates that COA. However, it is important to note, that Klein (1989, as 
cited in Wickens, Lee, Liu, and Becker, 2004) describes the RPD model as a “more 
refined description” of the various components of Rassmussen’s (1983, 1986, 1993) 
Skill, Rule, and Knowledge (SRK) model (p. 173). However, while the aforementioned 
models describe the process behind decision making, they do not adequately add the 


important aspect of perception. 
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Figure 1. Klein’s Recognition Primed Decision Model (1993). 
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2. Perception 


Endsley (1995) discusses perception in terms of situation awareness (SA). She 
defines SA as the “perception of the elements of in the environment within a volume of 
time and space, the comprehension of their meaning and the projection their status to the 
near future” (Endsley, 1995, p. 36). Consequently, the information presented to the 


commanders can be radically altered by their situational perception. 


To remedy this gap, Wickens et al. (2004) offers an integrated, adaptive decision 
making model (see Figure 2). Although the model effectively integrates RPD, SA, and 
SKR, it fails to address the symbiosis of man and machine in the battle space (i.e., the 
advent of technology discussed in Chapter II.A.1 has made the two parties mutually 
dependent of each other). Consequently, when discussing the role of the warfighter in 
“sensemaking,” one must apply a model that incorporates both man and machine. The 
model that best portrays this necessary relationship is the Dynamic Model of Situation 


Cognition (DMSC) (Shattuck & Miller, 2006; Miller & Shattuck, 2006). 
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Figure 2. Wickens et al. Integrated Model (2004) . 
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C. DYNAMIC MODEL OF SITUATION COGNITION 


The DMCS represents the integration of man and machine in the battle space (see 
Figure 3). It shows the processes associated with human-machine interaction when 
operating in complex, fast-paced, and uncertain environments. Additionally, it describes 
the perception, or lens, of the warfighter when making critical, tactical decisions. As 
described in Chapter I.A, because individuals are flawed and unable to grasp all 
knowledge they may naturally interpret information differently. Lastly, it addresses the 
inherent uncertainty involved in cognitive reasoning when conducting real-world 


operations. 


1, Principles of DMSC 


DMSC contains two distinct and important principles: (1) ground truth will never 
be completely realized or recognized by the decision maker and (2) the machine or 
system cannot identify or articulate uncertainty through its medium. Both principles are 
the curse of the information technology age. Because society is obsessed with the idea of 
perfection, we strive to create no doubt or uncertainty in our world. We have been 
conditioned to assume the technology cannot be wrong; it is the antidote to fix our 
inherent flaws. Consequently, we allow ourselves to be drawn into the perfect storm, or 


as Miller and Shattuck (2006) profoundly call it, “the seduction of technology.” 


This is no more evident than in the military, where we often struggle to remedy 
the effects of uncertainty in the tactical, operational environment. Even worse, the 
physical representation of uncertainty continues to present an enormous challenge in the 
modeling and simulation discipline. Regardless, before we can study the effects of 
uncertainty on decision making, we must first identify how the uncertainty manifests in 
both the technology and human systems. Fortunately, DMSC answers this critical 


question. 
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Figure 3. | Dynamic Model of Situated Cognition (Shattuck & Miller, 2006). 


2. Uncertainty Emergence 


The DMSC shows the presence of uncertainty throughout the model. It appears as 
early as Oval 2, and influences the decision making process. As Shattuck and Miller 
(2006) note, “the specific comprehension achieved by the decision maker is a function of 
the data that have propagated through the model and the contents of the lens,” (p. 998). 
Consequently, Oval (5) and Oval (6) show uncertainty’s effect on the decision maker’s 
comprehension and projection. However, before we can discuss uncertainty in the context 


of the model, we must explore the ovals where it manifests. 


a. Oval 1 


Oval 1 represents ground truth; ground truth being the absolute, 


unequivocal truth as described by Descartes (1637). 


b. Oval 2 


Oval 2 is how accurately technology depicts ground truth. Technological 
accuracy is determined by the efficiency of applied sensors (e.g., if the sensors do not 
detect various units in the battle space, then the unequivocal representation of ground 


truth is lost) (Shattuck and Miller, 2006). Additionally, the data can be misidentified. 
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Shattuck and Miller suggest that in a perfect world Ovals 1 and 2 would be mirror 


images, thus allowing for the decision maker to view the world from “God’s eye.” 


c. Oval 3 


Oval 3 describes the transition from the sensors to the display. Shattuck 
and Miller (2006) note that current C2 systems (Blue Force Tracker, Tactical Tomahawk 
Weapons Control System) allow for decision makers to tailor their displays to view 
perceived important information. Consequently, some information may not be seen or 
processed. Additionally, faulty algorithms may misinterpret what is ground truth, thus 


depicting an inaccurate picture. 


d. Oval 4 


The transition from Oval 3 to Oval 4 represents the bridge between 
technology and the operator (hence, human systems integration). Oval 4 shows the data 
perceived by the decision maker. Shattuck and Miller (2006) describe this process in 
terms of passive or active input. Active input is a result of the decision maker requesting 
or “pulling” information, where passive involves the information that is not requested or 


“pushed” upon the decision maker (Shattuck & Miller, 2006). 


e. Oval 5 


Oval 5 represents the comprehension of the decision maker. Endsley 
(1995) (as cited in Shattuck and Miller, 2006) describes comprehension as multiple 
processes, including integration, analysis, and interpretation. Additionally, Oval 5 is the 
result of Lens B in the model. Lens B articulates several factors including personality 
states and traits, experience, and social culture that influence the comprehension of the 
decision maker (Shattuck & Miller, 2006). Elements of Lens B will be discussed in 
Chapter II.F.2. 
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f Oval 6 


Oval 6 embodies the idea of projection. Shattuck and Miller (2006) define 
projection as “the prediction of the decision maker,” (p. 998). At this point, decision 
makers have ascertained what they believe to be true, and in some cases, have transmitted 


their depiction or beliefs about the situation to other decision makers in the battle space. 


g. Amorphous Shapes 


The amorphous shapes surrounding Oval 5 and Oval 6 represent the 
various organizations or interpretations of the information (Shattuck & Miller, 2006). 
Although the same information may be presented to two different decision makers, their 
interpretation of the information may be slightly or very different. Consequently, the 
decision makers may formulate different mental models of the situation (Shattuck & 
Miller, 2006). Additionally, the enlarging of the amorphous shapes from Oval 5 to Oval 6 
depicts the idea that there is more uncertainty involved in projection vice comprehension. 
A prediction of the future is merely one possible outcome of numerous possibilities for 
the future; the farther an individual predicts into the future, the more uncertainty will be 


present (Shattuck & Miller, 2006). 


3. Feedback Loops 


To curtail uncertainty in the DMSC, Shattuck and Miller (2006) emphasize the 
iterative, dynamic process inherent in the model. Feedback loops depict decisions or 
reorientations by the decision maker (see Figure 4). Thus, the individual can mitigate 
uncertainty by constantly updating their cognitive approach or the technologies’ output. 
Unfortunately, in most cases, the operator cannot simply “tag” an object as uncertain or 
ambiguous. However, they can inform other operators of the incorrect information being 


displayed. This leads to the next important point in terms of uncertainty. 
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Figure 4. Dynamic Model of Situated Cognition Feedback System (Shattuck & Miller, 
2006). 


Because NCW relies on a shared picture, or common operating picture (COP), the 
propagation of uncertainty in the battle space can be disastrous regardless of the echelon 
of command (i.e., uncertainty experienced by a flag officer or junior enlisted soldier can 
have equal effect). This is why the recent wars in Afghanistan and Iraq have focused on 
young soldiers making real time decisions with significant consequences. However, the 
impact of uncertainty up and down the chain of command is a pervasive issue and will 


need to be explored in future research. 


The present research examined tactical decisions made by middle echelons 
commanders (e.g., company commanders (CO CDR)). By doing so, the researchers 
established a basic understanding of the role of uncertainty in their decisions. More 
importantly, they examined how uncertainty influences the time it takes to make a 


decision and how individual differences help or hinder their responses. 


But before we can begin to study uncertainty, we must (1) define uncertainty, and 
(2) describe the individual differences that influence our decisions under uncertain 
conditions. While these tasks seem simple the answer is as ambiguous as the word being 


examined. 
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D. DEFINITION OF UNCERTAINTY 


Researchers continue to argue the definition of uncertainty. In capturing the mood 
of the debate, Downey and Slocum (1975, p. 562, quoted in Milliken, 1987, p. 134) state 
the overabundant use of the term “uncertainty” has made it “‘all too easy to assume that 
one knows what he or she is talking about.” Regardless, Lipshitz and Strauss (1997) 
propose uncertainty “in the context of action is a sense of doubt that blocks or delays 
action” (p. 150). Additionally, Lipshitz and Strauss distinguish uncertainty from 
ambiguity. Paraphrasing Hogarth (1987), the researchers describe ambiguity as “lacking 
precise knowledge about the likelihood of events” (Lipshitz and Strauss, p. 150). 

Conversely, Cohen and Freeman (1996) use a quantitative approach to 
characterize uncertainty. The researchers define uncertainty as “probabilities other than 
zero” (Cohen & Freeman, p. 179). Additionally, Conrath (1967) constructed a one- 
dimensional scale using the terms “certainty” and “uncertain” as the two extreme 
opposite points along the continuum. However, McCloskey and Klein (1996) go beyond 
defining the term, and identify four categories of uncertainty: (1) missing information, (2) 
unreliable information, (3) complex information, and (4) ambiguous/conflicting 
information. 

McCloskey and Klein (1996) define information as ambiguous or conflicting “if 
there is more than one reasonable way to interpret it” (p. 195). However, the researchers 
advocated a different approach. Rather than keeping ambiguous/conflicting information 
as a pair, they separated them into two different entities; ambiguous and conflicting. 
Then, from the pilot studies outlined in Chapter I.A, they adjoined missing and 
ambiguous. Additionally, the researchers viewed complex information as product of the 
other defined categories of uncertainty (i.e., the more uncertain a situation becomes, the 
more complexity involved). The modification of the categories of uncertainty reduced the 
number of categories to three. Finally, the researchers added the category of baseline to 
establish situations where little or negligible uncertainty appeared. Thus, the researchers 
viewed the categories of uncertainty as the following: (1) ambiguous/missing 
information, (2) conflicting information, (3) unreliable information, and (4) baseline 


information. 
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E. CATEGORIES OF UNCERTAINTY 


The categories of uncertainty provide structure for examining its affects in the 
natural world. However, before we can adequately study the levels of uncertainty, we 
must explicitly define the four terms. However, this research will only address three of 
the categories of uncertainty (ambiguous/missing, conflicting, and baseline). Examples 


are used to illustrate the differences between the aforementioned terms. 


1. Ambiguous/Missing Information 


Ambiguous/Missing data represents information that may be unavailable, but if 
available may be unclear or not critical to the decision. In other words, the decision 
maker may have information that is unnecessary to make an adequate decision, or may 
possess information that is very muddled and not explicit. Furthermore, the data may 
result in a choice between numerous alternatives. This last statement coincides with 


McCloskey and Klein’s (1996) interpretation of complex information. 


2. Conflicting Information 


Conflicting data represents multiple sources of information that is contradictory in 
nature. Mathematically this is expressed by x is true, y is true, but x equals y is not true. 


An example is provided later in this chapter. 


3. Unreliable Information 


Unreliable data consists of information where the source possess very little or no 
credibility (McCloskey & Klein, 1996). The decision maker does not trust the source, or 
feels the source contains low fidelity. This type of information is the link between the 
technological and warfighter sides of the DMSC, because it is dependent on whether the 
operator views the system or the human as reliable (Shattuck & Miller, 2006). 


4. Baseline Information 


Baseline data represents information that is available, understood, and possesses 


negligible uncertainty. 
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5; Distinguishing the Definitions 


Crafting understandable and distinguishable definitions for the categories of 
uncertainty was not a trivial task. Crafting suitable definitions involved discussions, 
much iteration, and multiple pilot studies to truly capture the essence of each category. 


Historical examples that illustrate the differences are provided below. 


Ambiguous/missing information was presented to French General Napoleon 
Bonaparte at the Battle of Waterloo. On June 18", 1815, the French army prepared to 
destroy the Duke of Wellington’s English force. Days earlier, Napoleon defeated the 
Prussian Armies at Ligny. However, his victory over the Prussians came with a price. 
The Prussians headed east, away from French scouts. Without intelligence on important 
information such as location, heading, speed of advance, casualties, or intentions of the 
Prussian armies, the French general was placed in a difficult and frustrating position. 
Napoleon wanted to attack the left flank of the British forces, but the idea of the Prussian 
Army somewhere to his right flank was not only dangerous but unnerving (Keegan, 
1978). The reality of the ensuing battle left the “genius” of Napoleon marginalized. The 
ambiguous nature of the Prussian whereabouts led Napoleon to conclude that a frontal 
attack was his best option (Keegan, 1978). Consequently, he was defeated, and forced to 


surrender. 


In contrast, on October 1962, conflicting information was presented to President 
John F. Kennedy. On October 14, 1962, U.S. intelligence took photographs of Soviet 
Union nuclear, ballistic missiles in Cuba. The placement of nuclear weapons just 
hundreds of miles off the coast of Florida placed U.S. national security at grave risk. 
President Kennedy chose not to inform the Soviets of his discovery. Instead, the 
President and his advisor debated over alternatives for an appropriate response. However, 
on October 13 President Kennedy met with the Soviet Minister of Foreign Affairs, 
Andrei Gromyko. When asked about the nature of Soviet weapons in Cuba, Soviet 
Minister Gromyko told President Kennedy that the Soviet Union weapons were of a 
defensive nature (Brugioni, 1991; Hilsman, 1996). While President Kennedy most likely 


immediately dismissed Gromyko’s claim, the Soviet Minister of Foreign Affairs provided 
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the President with a classic example of conflicting information; either the Soviet Union 
did or did not have offensive weapons in Cuba. Consequently, President Kennedy had 


two alternatives that were contradictory in nature. 


F. UNCERTAINTY TAXONOMY 


The uncertainty taxonomy was developed to better understand the factors that 
create and reason with uncertainty in the battle space. Like the DMSC, the uncertainty 
taxonomy examines the factors that influence decision making under uncertainty (see 
Figure 5). While human cognition is responsible for comprehension and projection of 
uncertainty, multiple factors, including technological systems, communication, and 
individual differences shape the conditions in which uncertainty can arise. Consequently, 
the uncertainty taxonomy provides means for classifying uncertainty types and for 
suggesting viable areas of research to determine the impact of the uncertainty types on 
performance. In order to better explain the relevance of the uncertainty taxonomy, the 
researchers divided into two parts: (1) information flow (IF) and (2) individual difference 
factors (IDF). 

Like the DSMC, information flow focuses on the projection of information or 
sources of information in the battle space. This structure parallels McCloskey and Klein’s 
(1996) levels of uncertainty. The IF is not just limited to humans, but also machines. 
Conversely, IDFs focus solely on the human’s abilities and cognition. This coincides with 
Shattuck and Miller’s (2006) Lens A and Lens B in the DMSC. However, it is important 
to note, trust is the interlinking variable (unreliable information) in both the IF and IDF 
branches (1.e., how well does an individual trust one’s self, in addition to, trusting the 
technology”). 

The ultimate goal of the uncertainty taxonomy is to analyze these associations 
with respect to their influence on decision making. Consequently, the IDF section 
discusses the influence of experience and personality, and examines how individuals cope 


with uncertainty. 
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Figure 5. Uncertainty Association Taxonomy. 


1. Information Flow 


IF is critical in military operations. Whether at the team performance level or 
individual performance level, the efficiency and speed of IF can determine the outcome 
of the engagement. Without adequate IF, a unit or commander may be paralyzed or 
unable to make critical, time-sensitive decisions due to the inconsistency or unavailability 
of information. Consequently, the interactions of several factors contribute to the 


aggregate effectiveness of IF in conducting continuous operations. 


a. Information Flow Interaction 


The interaction of trust, data, sources, uncertainty, and communication is 
complex. Instead of viewing these terms as abstract, independent ideas, the research 
addressed them using a systems approach (i.e., rather than considering these terms 
individually, the present research addressed their interaction). First, uncertainty can result 
from the interaction of data and communication. Data can be in the form of 
ambiguous/missing information, conflicting information, unreliable information, or 
baseline information. Additionally, the communication of data can be the aforementioned 
categories of uncertainty. For example, if an Officer of the Deck (OOD) witnesses on 


radar two contacts off the starboard bow, and a sailor looking at the same input from a 
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different display reports three contacts off the starboard bow, then the communication of 
the data, not the raw data itself, is uncertain. Conversely, if the sailor witnesses three 
contacts off the starboard bow from a different input source on a different screen, then the 


raw data, not the communication of the data are uncertain. 


Second, sources can be uncertain (i.e., a source, whether a system or 
human can be conflicting, unreliable, or ambiguous/missing). In addition to being 
uncertain, sources can possess the characteristic of trustworthiness. Kramer (1999) 
defines trust as “a state of perceived vulnerability or risk that is derived from individual’ s 
uncertainty regarding the motives, intentions, and prospective actions of others on whom 
they depend” (p. 571). In other words, trust is an individual’s acceptance of the 


consequences for another’s (man or machine) actions. 


Zand (1997) describes IF as a characteristic of trust. He discusses the 
effects of “[predisposed] beliefs” and “short-cycle feedback” loops (p. 94). He suggests 
that “people build a data bank, gathering impressions of other’s trustworthiness...these 
impressions are short-cycle feedback loops; they confirm or disconfirm” an individual’s 
trustworthiness (p. 94). Additionally, Yanik and Kleinberger (2000) suggest that decision 
makers are constantly correlating the quality of the advice given to the individual who 


gave them the advice. 


Even in the absence of feedback loops, decision makers, as reflected in 
their advice weighing policies, are aware of the quality of the advice received (Yanik & 
Kleinberger, 2000). The researchers offer two suggestions for the phenomena: (a) 
decision makers “relied on ranges of estimates” to determine accuracy, and (b) decision 
makers may have performed “plausibility checks,” and “recognized that a particular 
estimate was out of bounds” (Yanik & Kleinberger, pg. 270). Not only does this suggest 
that trust relies on the acceptance of the consequences for another’s actions, but it also 


shows that prior knowledge or experience often guides decision makers. 


22 Individual Difference Factors 


While IF is critical to C2 at the operational level, IDFs translate into individual 


performance. IDFs consist of several elements, including both intrinsic and extrinsic 
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factors. Personality characteristics, knowledge, cognitive processing, social interaction, 
training, and experience all contribute to a decision maker’s ability to perform (Miller, 
N.L., Crowson, J.J., & Narkevicius, J.M., 2003). Consequently, researchers must 
examine human behavior under various, unpredictable conditions to fully grasp and 


eventually predict human behavior (Miller et al., 2003). 


How an individual responds to adversity, risk, or uncertainty may play a vital role 
in their choice of alternatives. This was the key question of the researchers’ work, and for 
the scope of the study, they examined how individuals cope with uncertainty and how it 
related to their decision times. In the present study, the researchers focused on three 
aspects of specific IDFs: (a) operational experience, (b) personality characteristics (as 
measured by NEO Five Factor Indicator (NEO-FFI)), and (c) the ability to cope with 


uncertainty (as measured by Uncertainty Response Scale (URS)). 


a. Operational Experience 


An individual’s competence is derived from his or her ability to transform 
prior experience to decisive action. Klein’s (1997) research using the Recognition Primed 
Decision (RPD) model (Klein, Calderwood, & Clinton-Cirocco, 1985) supports this 
claim. Klein (1997) found that decision makers, in dynamic environments, mostly rely on 
past experience when deciding on their course of action. Kaempf, Wolf, Thordsen, and 
Klein (1992, as cited in Klein, 1997, p. 289) examined 78 cases of decision making on 
U.S. Navy AEGIS Cruisers, and estimated that in 78% of the cases the “decision maker 
adopted the course of action (COA) without any deliberate evaluation” (p. 289). This 
suggests that decision makers do not weigh alternatives, but rather choose based on prior 


experience (Kaempf et al., 1992; Klein, 1997). 


St. John, Callan, Proctor, and Holste (2000) used Tactical Decision Games 
(TDG) to measure how uncertainty affects decision making among U.S. Marines. The 
researchers found that when facing increased uncertainty, the majority of less 
experienced Marine officers chose a “wait and see” option, while the more experienced 
officers did not (St. John et al., 2000). This study coincides with the findings of Serfaty, 
MacMillan, Entin, and Entin (1997), who found that participants with greater expertise in 
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a given field provided a more detailed COA compared to participants with less expertise. 
They concluded their findings “[were] consistent with the idea that experts are able to 
draw on previous experience to generate a more complete schema for the tactical 


situation” (Serfaty et al., p. 242). 


In this study, experience is defined as operational experience. Operational 
experience is defined as months in an operational theatre (e.g., Iraq & Afghanistan). 
However, the researchers are confident that while all the participants received specialized 
training for their occupations, they also received common basic training. Consequently, 


each participant had enough training and experience to be evaluated in the study. 


b. NEO-FFI 


Experience is the most obvious extrinsic factor, but personality is also 
important when examining decision making under uncertainty. The NEO Personality 
Inventory has been the standard for research in personality traits and characteristics. 
However, unlike the NEO Personality Inventory, the NEO-Five Factor Indicator (NEO- 
FFI) only addresses the five major elements of personality, and not the different facets 
that accompany each (Costa & McCrae, 1992, & Hannen, 2007). For this study, the 
NEO-FFI was used to look at the relationship between personality and uncertainty. The 
five major elements of the NEO-FFI are the following: 

e Openness (O) 

e Agreeableness (A) 


e Conscientious (C) 
e Neuroticism (N) 
e Extraversion (E) 


(1) Openness (O) refers to individuals who are “open to 
experience” and “curious about both inner and outer worlds,” (Costa & McCrae, 1992, p. 
15). Additionally, individuals who score high on O are not usually conservative in 
thought, but rather unorthodox thinkers with creative imaginations (Costa & McCrae, 
1992). Under uncertainty, the researchers would expect a decision maker to not 


necessarily respond slower or faster, but rather adopt a clever, inventive decision. 
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(2) Agreeableness (A) refers to individuals who are 
“fundamentally altruistic” and “sympathetic to others and eager to help them,” (Costa & 
McCrae, 1992, p. 15). However, Costa and McCrae (1992) are quick to note, that 
agreeableness is not a preferred characteristic in occupations where one’s own self- 
interests may be at risk (e.g., military, police, firefighter). Consequently, in a sample 
pool involving military service members, one would not expect respondents to score high 
on A. Under uncertain conditions, the researchers would expect individuals who score 


high on A to respond quickly with sometimes irrational, if not brash, decisions. 


(3) Conscientiousness (C) examines whether an individual is 
“purposeful, strong-willed, and determined,” (Costa & McCrae, 1992, p. 16). 
Additionally, Costa and McCrae (1992) suggest conscientiousness is somewhat 
analogous with character (i.e., individuals who possess high C scores are usually 
considered credible and in high moral stature). The researchers would expect individuals 
who score high on C would not necessarily respond faster or slower, but be very aware of 


both ethical and moral elements of their decisions, not to mention, self-reflective. 


(4) Neuroticism (N) refers to the emotional stability of an 
individual (Costa & McCrae, 1992). Moreover, an individual who possess low N scores 
will usually be “calm, even-tempered, and relaxed,” (Costa & McCrae, 1992, p. 15). 
More importantly, they are able to deal with stressful environments (Costa & McCrae, 
1992). Military officers often perform in complex, operational environments, therefore 


they would be expected to score low on N. 


(5) Extroversion (E) examines whether or not an individual is 
“assertive, talkative, or active,” (Costa & McCrae, 1992, p. 15). Additionally, it explores 
whether an individual socializes in larger groups or even likes people in general (Costa 
&McCrae, 1992). Faced with uncertainty, individuals who score high on E would be 
quick, even aggressive, to make a decision. However, it is unknown whether their 


decision would be ill-thought or carefully calculated. 


In conclusion, the NEO-FFI may provide insight into decision 


making under uncertainty. The different elements of personality may lend to increased 


ZS 


resolve when faced with uncertainty in a time-sensitive, stressful environment. However, 
we must explore whether these characteristics are congruent with the methods or 


techniques used to cope with uncertainty. 


c. Coping with Uncertainty 


Although experience and personality characteristics may influence a 
decision maker’s choice of alternatives, perhaps another important measure may lie in 
how a commander responds emotional and cognitively to uncertainty. To explore this 
question, the Uncertainty Response Scale (URS) explores the effects of uncertainty from 
three, different perspectives: (1) emotional uncertainty (EU), (2) desire for change (DFC), 
and (3) cognitive uncertainty (CU) (Greco & Roger, 2001) (see Appendix A). 


EU is defined as “the degree to which an individual responds to 
uncertainty with maladaptive behavior” (Sutton, Cosenzo, & Pierce, 2004). Higher scores 
on EU indicate an individual does not adapt well to uncertain situations (Thomas, 2005). 
Interestingly, Greco and Roger (2001) found a significant correlation between N and EU 
(r = 0.56, p < 0.01). Additionally, EU correlated significantly with Kirton’s (1981) 
Tolerance to Ambiguity (TOA) (r = 0.18, p < 0.01). Consequently, we suggest 
individuals who score high on EU and N erratic response times in an uncertain, tactical 


environment. 


DFC is classified as “the enjoyment of uncertainty, novelty, and change,” 
(Greco & Roger, 2001, p. 525). Higher scores on DFC indicate that an individual enjoys 
uncertainty or the unknown (Thomas, 2005). Greco and Roger (2001) found significant 
correlations between sub-elements of E (e.g., impulsivity and sociability). The 
researchers found a moderate association between DFC and impulsivity (r = 0.37, p < 
0.01) and DFC and sociability (r = 0.23, p < 0.01). Unfortunately, these results give no 
insight into whether individuals with high E scores will perform adequately under 


uncertain conditions. 


CU is described as the “degree to which an individual prefers order, 
planning, and structure in an uncertain environment,” (Thomas, 2005, p. 35). Higher 


scores on CU indicate that an individual desires control or organization in uncertain 
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situations (Thomas, 2005). Greco and Roger (2001) found a significant correlation 
between CU and TOA (r = 0.37, p < 0.01). Additionally, the researchers discovered a 
significant correlation between CU and N (r = -0.20, p < 0.05). Moreover, the E sub- 
element of impulsivity was significantly correlated to CU (r = -0.25, p < 0.01). As 
previously mentioned, military officers often operate in highly structured, yet complex, 
environments. Consequently, these results suggest officers scoring high in cognitive 


uncertainty may perform well when faced with uncertainty. 


These significant correlations provide evidence of the important 
relationship between personality characteristics and an individual’s method for coping 
with uncertainty. The URS has both high internal consistency reliability (EU = 0.91, CU 
= 0.87, DFC = 0.87) and test-retest reliability (EU = 0.79, CU = 0.80, DFC = 0.86), thus 
procuring credibility for future research. More importantly, Greco and Roger (2001) 
provide an excellent basis for inferring a relationship between uncertainty and personality 
characteristics. Their work has implications for the present research in tactical decision- 


making under uncertainty. 


G. CRITICAL DECISION MAKING METHOD 


Although the decision maker’s choice of alternative is important to understanding 
his or her cognitive reasoning, it is not the focal point of this study. Instead, research 
efforts were directed toward discovering the impact of uncertainty on the commander’s 
decision. This paradigm shift is the result of two objectives. First, the study of uncertainty 
was undertaken to determine “why and how” uncertainty affects the commander’s lens. 
Secondly, the study was focused in discovering where uncertainty occurs in the tactical 


environment. 


Until this point, this literature review has explored the “why” and “how” question, 
but not adequately addressed the “where.” Consequently, before we can develop a 
methodology to test uncertainty in decision making, we must expose and unearth 
uncertainty in the battle space, (i.e., in the ovals in the DMSC) (Shattuck and Miller, 
2003). In simpler terms, we must find a process to elicit uncertain situations in the 


operational world. 
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Klein et al. (1989) developed the elicitation process of Critical Decision Method 
(CDM). The techniques of examining cases based on non-routine events by cognitive 
probes and maintaining a semi-structured approach allow researchers to uncover the 
methods and reasoning behind an individual’s decision. These techniques follow a top- 
down approach to determine how and why a decision maker chose an alternative. More 
importantly, as Klein et al (1989) points out, CDM provides a method for evaluating 
performance to better assist in identifying training requirements. Consequently, CDM is a 
powerful tool in reconstructing naturalistic events in the hopes of creating more realistic 


and efficient models for training simulations. 


While Klein’s interest lies in the decision maker’s cognitive reasoning, CDM was 
modified in this study to elicit the presence of uncertainty in the battle space. Developed 
by Dr. Lawrence Shattuck, the modified CDM follows the outline presented by Klein et 
al (1989), but instead uses cognitive probing techniques to unearth the presence (i.e., the 
“where”) of uncertainty in the participant’s recollection of events (see Appendix B). 
Consequently, the decision becomes secondary in the process. This alteration allows the 
researchers to categorize the witnessed uncertainty using the definitions provided in 
Chapter II.F. Moreover, it enables a clear examination of the chronological events prior 


to and after uncertainty appears in the equation. 


H. HYPOTHESES 


This literature review has uncovered many important questions in the field of 
tactical decision-making under uncertainty. The questions have been narrowed to those 


deemed most important. The alternative hypotheses generated from those questions are 


listed below. 

e Ha,: There is a significant difference in the overall total time with respect 
to the three categories of uncertainty. 

° Haz: There is a significant, positive correlation between emotional 
uncertainty and overall total time for ambiguous/missing information and 
conflicting information. 

e Ha: There is a significant, negative correlation between cognitive 


uncertainty and overall total time for ambiguous/missing information and 
conflicting information. 
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Ha3: There is a significant, negative correlation between operational 
experience and overall total time for ambiguous/missing information and 
conflicting information. 
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Hl. METHOD 


Research began in January 2007. Over the past 18 months three small scale pilot 
studies were conducted. The results shaped the methods used in the main study. This 
section contains short, concise descriptions of the three pilot studies, in addition to, 


lessons derived from their results. Finally, the main study method will be outlined. 


A. PILOT STUDIES 


This section addresses the pilot studies conducted by the researchers. Each pilot 
study focused on the research questions outlined in Chapter II.C. It is important to note, 
the latter pilot studies built off the successes and failures of the preceding studies. 


Consequently, they will be discussed in chronological order. 


1 Fe Pilot #1 - Ambiguous Words 


In the first pilot study, a research team, consisting of Naval Postgraduate School 
(NPS) students, attempted to: (1) operationally define ambiguity, and (2) design a study 
to test its effects on decision-making with respect to the time it takes to make a decision. 
After a review of the literature, the team began developing models to understand 
uncertainty. The team considered the research of McCloskey and Klein (1996) as the 
most robust definition of ambiguity and uncertainty. Their work provided the basis to 


design an experiment examining uncertainty at the fundamental level. 


a. Participants 


Thirty-two NPS students (30 male, 2 female) volunteered to participate in 
the initial pilot study. All participants were commissioned officers, who were in the 
process of earning their master’s degree. Of the group, twenty-nine participants had 20/20 
vision, while only three reported being colorblind. The colorblind participants were 


omitted from the results. No compensation was given for their participation. 
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b. Method 


The participants were shown numerous Microsoft PowerPoint slides with 
a word in pink letters (see Appendix C). The words were drawn from the American 
National Standards Institute (ANSI) Standard S3.2-1989, Method for Measuring the 
Intelligibility of Speech over Communications System (2005). Transposed over the pink 
letters was a rectangular pink square, which was adjusted for resolution to create different 
severities of ambiguity (high, medium, low). The participants were given four choices 
and asked to match the word shown on the slide to a word on their answer sheet. 
Confidence was measured by survey using a Likert scale. Time to decision (time to 
decision) was measured by an electronic timer embedded in the software. Participant 


accuracy was determined by grading the correctness of their responses. 


Cc. Results 


The researchers analyzed three separate aspects of the data using a within 
subjects design. First, using ANOVA, the researchers found a significant difference 
between the three severities of ambiguity (high, medium, low) with respect to time to 
decision (F (2, 320) = 76.20, p < 0.001). Next, the researchers used non-parametric 
measures (contingency analysis) to determine the relationship between confidence and 
ambiguity severity level. As expected, they found a higher percentage of “less confident” 
participant responses when presented with the “high” severity of ambiguity compared to 
medium and low severities. Finally, the researchers analyzed the accuracy rates of the 
three severities of uncertainty. They discovered the following: High = 85.6%, Medium = 


96.9%, Low = 99.7%. 


d. Discussion 


Overall, the study provided an initial approach to the topic of uncertainty 
and ambiguity. The results suggest that ambiguity severity affects response time. 
Additionally, the results infer that accuracy and confidence are degraded by the presence 


of ambiguity. Unfortunately, the study did not address whether ambiguity and uncertainty 
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were, in fact, synonymous terms. If ambiguity and uncertainty are not synonymous, then 
how are they different (or differentiated)? Is one subsumed by the other? Are there other 


categories that are separate but related? 


While these questions were important to answer, another need slowly 
emerged from the study. When the results from the first pilot were presented to military 
officers and civilian officials, they questioned the relevance and realism of the stimuli 
(i.e., how does the study link to the ambiguity seen by commanders on the battlefield?). 
Consequently, the researchers returned to the NDM literature to find examples (solutions) 
of real-world, operational environments. Additionally, the results and conclusions from 
the first pilot study moved the researchers to develop two distinct goals for the 
subsequent pilot study: (1) to develop a better operational definition of uncertainty, and 


(2) to design a study that incorporates a relevant military setting. 


2. Pilot #2 - Operational Intelligence 


For the second pilot study, the research team expanded the scope of research. 
McCloskey and Klein (1996) suggest that four categories of uncertainty in the realm of 
uncertainty. They described them as follows: (1) missing information, (2) complex 
information, (3) ambiguous/conflicting, and (4) unreliable information (McCloskey and 


Klein, 1996). 


However, the researchers’ experience in tactical environments led them to believe 
ambiguous/conflicting were two separate categories. Additionally, the idea of complex 
information seemed too abstract to operationally define given a relatively short timeline. 
Moreover, complexity seemed inherent in uncertainty (i.e., complexity was proportional 
to uncertainty). Consequently, the research team opted to evaluate three categories: (1) 


missing information, (2) conflicting information, and (3) ambiguous information. 


a. Participants 


Thirty-four participants from NPS, which included 14 naval officers, 5 


Marine Corps officers, 6 army officers, 5 foreign military officer, and 4 civilians 
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participated in the second pilot study. They ranged from two to twenty years of military 


service and were not compensated for their participation. 


b. Method 


The participants were placed in an operational situation. They were 
briefed on a hostage crisis involving several senior U.S. State Department officials, 
including the Secretary of State. The experimenters explained that intelligence agents had 
placed three GPS transmitters on key hostage personnel in preparation for a hostage 
rescue mission. The three GPS transmitters each displayed a single shape (circle, triangle, 


square) on a 3 x 3 targeting grid (see Figure 6). 


Ne eee 
j ae 


Q. WHERE IS THE CIRCLE? 





Figure 6. —_ Pilot Study #2 Situational Display. 


Each participant was shown 20 slides that contained the shapes under the 
four categories of uncertainty (conflicting information, baseline information, ambiguous 
information, missing information) (see Appendix D). Each slide asked a question relating 


to the location of a shape. Each scenario was displayed for a total of twenty seconds, and 
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the question appeared with ten seconds remaining. The participant was then required to 
analyze the scenario and answer the question to the best of his or her ability. A ten second 
interstitial slide was displayed to allow the individual to indicate his or her certainty 
response level on a corresponding Likert scale. The Likert Scale ranged 1 to 9 with 1 


corresponding to “not at all certain” and 9 corresponding to “100% certain.” 


c. Results 


The researchers collected three sets of data: (1) certainty response level 
(Likert Scale), (2) grid location choice, and (3) NEO-FFI. Non-parametric statistics were 
used to analyze the certainty response levels, while Pearson correlation coefficient was 
used for the NEO-FFI and response time data. The researchers found a significant 
difference between the four categories of uncertainty (see Table 1), but did not find a 


significant correlation to any of the five factors of the NEO-FFI. 


Ls Miss - Base _| Con-Base | Amb-Base | Con-Miss | Amb—Miss | Amb-Con 


Z 


Asymp. Sig. 
(2-tailed) 




















Table 1. Sign Test for Categories of Uncertainty. 


The study suggests that individuals differ in their degree of certainty with 
respect to the category of uncertainty. Individuals appear to be less certain when faced 
with ambiguous information than with conflicting information (i.e., individuals would 
rather have a choice between two alternatives vice a choice amongst many or unknown 


alternatives). 


d. Discussion 


The results from Pilot Study #1 and Pilot Study #2 produced another 
important question. If the categories of uncertainty contribute to significant differences in 


certainty response levels and if different severities for ambiguity lead to significant 
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differences in time (see Chapter IIL.A.1), then will different categories of uncertainty 
cause significant differences in time? To answer this question the research team 


developed realistic tactical vignettes for each category of uncertainty. 


3. Pilot #3 - Tactical Decision Environments 
a. FT Leavenworth, KS 


(1) Participants. Interviews were conducted at Command 
General Staff College (CGSC), FT Leavenworth, KS, using the modified version of 
Klein’s (1996) CDM (see Chapter I.G.). The participants consisted of 20 U.S. Army 
officers (2 females and 18 males with a mean age of 36.9 years). Nineteen participants 
held the rank of O-4 or Major, while one participant held the rank of O-3 or Captain. The 
participants had an average of 16.0 years of military service and 13.1 years of 
commissioned service. These averages suggest many participants served in the enlisted 
ranks prior to earning a commission. Additionally, the participants had an average of 15.5 
months of deployment time post September 11", 2001 (ie., the participants had a 


creditable amount of operational experience in the War on Terror) 


(2) Method. To begin the knowledge elicitation, the 
researchers used the following paragraph as their primary inquiry: 
Recall a tactical situation during a recent deployment to a combat zone in 
which you were confronted with uncertainty and had to make a decision. 
Uncertainty can be described simply as a sense of doubt that blocks or 


delays action. There are no right or wrong answers and we are in no way 
evaluating your performance or the decisions you made. 


Each participant was asked to provide two situations or scenarios. 
Additional follow-on questions used for data elicitation can be found in Appendix B. 


These questions were used to explore the uncertainty in the described situation. 


The researchers analyzed the scenarios described by the 
participants. The majority of the incidents of uncertainty recalled were categorized as 
ambiguous. Two scenarios were selected and modified for the data collection at NPS. 
The researchers created four versions of each scenario (i.e., conflicting, missing, 


ambiguous, and baseline) for a total of eight situations. 
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b. Naval Postgraduate School 


(1) Participants. The participants at NPS consisted of 17 
officers, including one female and sixteen males with an average age =32.5 years. Nine 
participants were U.S. Navy officers, three were U.S. Army officers, one was a U.S. Air 
Force officer, one was a Canadian Air Force officer, and three participants were Turkish 
Air Force officers. Six participants held the rank of O-4, 10 held the rank of O-3, and one 
participant held the rank of O-2. None of the participants were compensated. 


(2) Method. Each participant saw a total of eight slides. They 
were given two pieces of information: (1) operational order (OPORD) information and 
(2) additional information (AI). The OPORD provided data on the situation, enemy 
forces, friendly units, terrain, and tactical situation. The additional data provided real time 
intelligence, including enemy movement, enemy killed, and situational reports 


(SITREPS). 


The OPORD first appeared followed by the AI. All the information 
was present within 24 seconds of scenario commencement; thus, the participant had 
direct access to all information available to make a decision. Once a decision was made, 
the participant pressed the space bar, which recorded the total time from beginning to 
end. However, the total time was then subtracted by 24 seconds to produce time to 


decision. 


After the participant pressed the space bar, the interviewer, located 
in the same room, asked the participant to: (1) provide a brief description of the decision, 
and (2) list any contributing factors to the decision? The researchers took written notes of 
the participant’s responses. Digital voice recorders (DVRs) recorded each session. The 


quality and accuracy of their answers were not analyzed. 


(3) Results. The researchers analyzed the difference scores of 
time to decision for each category of uncertainty. The distributions of the difference 
scores of time to decision for each category of uncertainty violated the normality 


assumption, thus non-parametric tests were conducted for analysis. 
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First, the researchers conducted an analysis to determine whether 
or not a learning effect occurred between Situation | and Situation 2. Due to the small 
sample size, the researchers set alpha (a) at 0.10. They discovered a significant learning 
effect between Situation 1 and Situation 2 (p = 0.09). Consequently, the researcher did 
not analyze Situation one data. Instead, they analyzed Situation two data by taking the 
difference scores from each category of uncertainty and subtracting it from the baseline 
for each participant (e.g., ambiguous information time to decision — baseline information 


time to decision = delta (ambiguous information, baseline information)). 


The researchers found a significant difference between the 
difference scores for ambiguous information compared to conflicting information (p = 
0.097). Additionally, they discovered a significant difference for conflicting information 
compared to baseline (p = 0.073). The researchers did not find a significant difference 


between missing and ambiguous information (p = 0.52). 


(4) Discussion. The pilot study identified issues that needed 
reconciliation prior the final empirical study. First, the learning effect discovered between 
Situation | and Situation 2 revealed the need for several different, independent situations 
along with randomization. Consequently, the researchers incorporated this approach into 
their final study. Secondly, although “‘jointness” is often emphasized in the conduct of 
U.S. military operations, many of the participants were naval officers vice army officers, 
thus some terms were unfamiliar to the Naval officers. Therefore, the researchers 
determined that they needed a sample group that shared a common language and 
experience. Additionally, a list of acronyms was necessary to properly identify terms. 
Thirdly, the OPORD and AI was found to be too broad and generalized, thus being 
interpreted as “not real enough.” The researchers enlisted the help of a group of active 


duty or retired Army personnel to craft better, more detailed vignettes. 


Finally, and most importantly, the data suggested that ambiguous 
information and missing information were statistically the same, thus the researchers (as 
discussed in Chapter II.E) crafted new definitions and categories of uncertainty. These 


definitions became the final definitions for the CGSC empirical study. 
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B. CGSC EMPIRICAL STUDY 


1. 


Independent Variables 


Category of Uncertainty — ambiguous/missing information, conflicting 
information, baseline information. 


Block Number (Randomization) — Block 1, Block 2, and Block 3. 


Difficulty — mean difficulty rating calculated through inner rater reliability 
testing. 


Dependent Variables 


Overall Total Time (OTT) — the elapsed time from commencement of 
scenario to end of scenario. 


Situation time (ST) — the elapsed time from the commencement of the 
scenario to the when the participant requests the additional information. 


Additional time (AT) — the elapsed time from the time the participant 
requests additional information to when they finish reading the additional 
information. 


Situation/Additional Total Time (SATT) — the elapsed time from the 
commencement of the scenario to when the participant finishes reading the 
additional information. 


Time to Decision (TTD) — the elapsed time from when the participant 
finishes reading the additional information to when they inform the 
researcher they have made a decision. 


Time to Decision/Additional Total Time (TTDAT) — the elapsed time 
from when the participant requests additional information to when they 
inform the researcher they have made a decision. 


Uncertainty Response Scale Scores — the scores for the three categories of 
the URS (emotional uncertainty, cognitive uncertainty, desire for change 
uncertainty). 


Overall total time, cognitive uncertainty, and emotion uncertainty scores are the 


primary metrics of the study. The other variables are secondary metrics used to discover 


and explain differences in the outlined hypotheses. 


3. 


Participants 


The participants consisted of 28 students attending the CGSC, FT Leavenworth, 


KS. There were 28 males with an average age of 37 years and an average commissioned 


service time of 13 years. Twenty-seven participants held the rank of Major (O-4), and one 
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participant held the rank of Lieutenant Colonel (O-5). With respect to their branch type, 
14 served in Combat Arms (CA), 9 served in Combat Support (CS), and 5 served in 
Combat Service Support (CSS). 


Participants were treated in accordance with the “Ethical Principles of 
Psychologists and Code of Conduct” (American Psychological Association, 1992). The 
NPS Institutional Review Board (IRB) and CGSC Quality Assurance Office Survey 
Control approved the research methods used for the experiment; all participants signed 


voluntary consent forms and audio/video consent forms prior to participation. 


4. Materials and Apparatus 
a. Demographic Survey 


The demographic survey asked for information on age, military service, 
commissioned service, branch, and operation deployment time. Because the study 
focused entirely on U.S. Army officers, the questions were limited to the specified 


service. 


b. NPS Decision-Making Study 


From the lessons learned in the pilot study and the situations depicted by 
the CDM interviews, ten vignettes were crafted by members of U.S. Army Training and 
Doctrine Command, FT Leavenworth (TRAC-FLVN), and U.S. Army Training and 
Doctrine Command, Monterey (TRAC-MTY). The ten vignettes consisted of three 
ambiguous/missing information vignettes, three conflicting information vignettes, and 


four baseline information vignettes (see Appendix E). 


The vignettes were given to retired U.S. Army service members working 
at TRAC-FLVN for reliability screening. Each evaluator was given the definitions listed 
in Chapter IIE, and the ten vignettes. The evaluators were instructed to answer three 
questions: (1) what category best describes the uncertainty in scenario, (2) how difficult 
would it be to make a decision in this scenario, and (3) how difficult was it to classify the 


category of uncertainty? 
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After several iterations, the researchers achieved 90% reliability for 9 out 
of the 10 vignettes. Consequently, the researchers used those 9 vignettes for the study 
(e.g., (3) ambiguous/missing vignettes, (3) conflicting vignettes, and (3) baseline 
vignettes). Additionally, the researchers achieved high inter-rate reliability for the level 
of difficulty (« = 0.81). The level of difficulty was different for each scenario (see Table 
2), and discussed and analyzed in Chapter IV. The vignettes were counter-balanced and 
given randomly to the participants. 


Ambiguous/Missing 3 





Table 2. Scenario Difficulty Means for the Categories of Uncertainty. 


Each slide contained two segments of information: (1) situation 
information, and (2) additional information. The situation information contained 
information on mission, enemy size, friend forces, and environment. The additional data 
provided real-time, battle space intelligence. For all vignettes, the situation information 
and additional information possessed approximately the same amount of data (see 


Appendix E). 


The researchers collected data on five different time intervals: (1) overall 
total time, (2) time to decision, (3) situation time, (4) additional time, (5) situation 
time/additional time, and (6) time to decision/additional time. Further explanation of 


these times is discussed in Chapter III.B.2. 


Stopwatches were used as the primary means of recording the times. The 
secondary means were DVRs. Training “clickers” were used to make distinguishable 


sounds for data analysis. The vignettes were presented using Microsoft PowerPoint, and 
4] 


displayed on a 52 inch flat, plasma screen television. The study was conducted in Lewis 


and Clark Memorial Hall, 2™ floor, at CGSC FT Leavenworth, KS (see Figure 7). 








Figure 7. | Photograph of CGCS Study. 


c. Uncertainty Response Scale (URS) Survey 


As described in Chapter I.F.2, the URS explored the various ways 
individuals cope with uncertainty. The URS data were used to help explain differences in 


performance across participants. 


d. Operational Experience 


Operational experience was collected by the aforementioned surveys. 
Operational experienced was measured by months in Afghanistan, months in Iraq, and 
total number of months in Iraq and Afghanistan. Participants with no operational 


experience in both theatres were omitted from the study. 
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5, Design and Procedure 


The study was a within subjects design. Upon entering the room, participants 
signed an Informed Consent Form and an Audio/Visual Consent Form. The participants 
were seated at a desk approximately five feet from the television screen. Next, they were 
asked to read an instruction sheet aloud (see Appendix F). Then, the participants were 
shown a slide with various assumptions about the tactical environment (see Figure 8). 
Subsequently, the participants completed an example slide. Lastly, the participants were 
shown 9 slides and times were recorded as mentioned previously. Each scenario 


proceeded in the following manner: 


*"IMPORTANT*** 


- Youwill given atotal of 10 scenarios;1 | 
example scenario and 9 evaluated scenarios. 
* For all scenarios, the following is true: 


1) Your units are properly equipped and capable of 
accomplishing the mission. 


2) Ifyou are an advisor, consider your advice equal to 
decision. 


3) Your command climate allows you to make 
decisions at your | evel. 

4) The scenarios ave not connectedin any way. They 
are independent, so consider every decision made 
in each scenario as separate and distinct from any 
other scenario. 


Figure 8. Tactical Decision-Making Assumptions. 


First, the participants were shown the situation information and the time started. 
The participants read the situation data aloud. When they were ready to receive the 
additional data, they snapped the training clicker. The training clicker was used to 
facilitate the analysis of the DVR data after the experiment (i.e., the training clicker 
created a sound with greater amplitude than a normal speaking voice, thus making it 


easier to identify the various dependent variables associated with time). 
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When the participants snapped their clicker for the additional data two events 
occurred: (1) the experimenter logged their time (situation time) and (2) the additional 
data was presented to the participants. However, it is important to note, that the situation 


data remained displayed, so the participant had access to all presented information. 


Second, when the additional data appeared, the participant read the additional data 
aloud. When the participants finished reading the last word of the additional data, the 


experimenter logged their time. 


Third, when the participants were ready to make a decision, they were asked to 
snap their training clicker. Upon snapping their training clicker, the experimenter logged 
their time. Thus, a total of three times were logged, and used to calculate the various 


times noted (see Figure 9). 


Participant finishes reading 
Participant snaps clicker and additional information. 
receives additional 


information. Participant snaps clicker to 


indicate they have made a 


Time = 0.0 seconds | Ri 
= decision. 














Figure 9. = Tactical-Decision-Making Assumptions. 


Finally, the participant was asked to answer two questions: (1) what is your 
decision, and (2) how did you arrive at your decision? The participants’ answers were 


recorded on DVRs, and later transcribed for evaluation. 


Upon completion of the NPS tactical decision-making study, the participants 
completed two surveys. First, they completed the URS. The title of the survey was not 
listed on the paper. Next, the participant completed the demographics survey explained in 
Chapter II.C.4. To ensure all documents and exercises were completed, the experimenter 


completed and signed an interviewer checklist. 
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IV. RESULTS 


The results consist of four parts: (a) summary statistics for categorical uncertainty, 
(b) summary statistics for URS, (c) summary statistics for operational experience, and (d) 
statistical analysis for the associations between categorical uncertainty, URS, and 
operational experience. Descriptive statistics are provided in parts a, b, and c. Inferential 
statistics are used in all parts to analyze differences across the categories of uncertainty 
with respect to time, along with appropriate correlations between variables were 
necessary. Due to discovered differences in the recorded times, between the stopwatches 


and DVRs, the researchers opted to use the DVR times for statistic analysis. 


A. SUMMARY STATISTICS FOR CATEGORICAL UNCERTAINTY 


For description purposes, additional time, situation time, and time to decision will 
be referred to as the adjoined times. Adjoined emphasizes that these times follow a 
chronological order (i.e., they are linked by one ending and the next beginning). 
Consequently, situational time is followed by additional time which is followed by time 
to decision. Additionally, situation time/additional time and time to decision/additional 
time will be referred to as the interrelated times. Interrelated emphasizes that these times 
overlap, and that they both use additional time in their calculation. Moreover, overall 
total time will be referred to as the decision time. The decision time is the total 
summation of the adjoined times. This terminology was developed to simplify reporting 
the results. Overall total time serves at the variable of primary analysis. The interrelated 
and adjoined times are supplementary analysis that will be used to better explain the 


differences in overall total time. 


1. Descriptive Statistics for Categorical Uncertainty 


For the adjoined times, ambiguous/missing information had the largest mean for 
situation time and time to decision, while conflicting information had the largest mean for 
additional time (see Table 3). Additionally, ambiguous/missing had the largest standard 
deviation (SD) across all the adjoined times. For overall total time, ambiguous/missing 


had the largest mean along with the largest SD. 
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sae (SD = 56) 
14.3 (SD = 13.5) 
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76.4 (SD = 17.9 











Table 3. Categorical and Time Segment Means and Standard Deviations for CGSC 
Study (seconds). 


2. Inferential Statistics for Categorical Uncertainty 


The data violated the normality assumption, thus non-parametric analysis was 
conducted to compare the categories of uncertainty with respect to time. To evaluate the 
times for each participant, the researchers calculated the means for each participant in 
each category of uncertainty. Each participant had mean times for each category of 
uncertainty for each segment of time. The researchers used the times and difficulty scores 


to determine whether difficulty was a significant factor in determining response time. 


a. Decision Time 


For overall total time, the researchers discovered significant differences 
amongst two of the categorical pairs of uncertainty: (1) ambiguous/missing information 
was significantly greater than conflicting information (p < 0.01), and (2) 
ambiguous/missing information was significantly greater than baseline information (p < 
0.01). Conversely, the researchers did not find a significant difference between 


conflicting information and baseline information (p = 0.06) (see Table 4). 


Mean (OTT , Baseline)- Mean(OTT, Conflicting)-Mean(OTT, 
Mean(OTT, Ambiguous/Missing) 
Ambiguous/Missing )} 





Test Statistic -1 a -1 a 


Prob > [z| o0674 





Table 4. | Wilcoxon Sign-Rank Analysis for Overall Total Time. 
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The researchers found a significant, positive correlation between all the 
categories of uncertainty for overall total time: (1) ambiguous/missing information and 
conflicting information (r = 0.87, p < 0.01), (2) ambiguous/missing information and 
baseline information (r = 0.84, p < 0.01), and (3) conflicting and baseline information (r 
= 0.8, p < 0.01) (see Figure 10). Hence, if a participant responded quickly to one category 


of uncertainty, he/she would respond quickly to the other categories of uncertainty. 
175 
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Figure 10. Participant Mean Overall Total Time for the Categories of Uncertainty. 


Regression analysis revealed that scenario difficulty was a significant 
predictor for ambiguous/missing overall total time (p < 0.01), but not a significant 
predictor for conflicting overall total time (p = 0.99) and baseline overall total time (p = 


0.95) (see Figure 11). 
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Ambiguous/Missing —— 
Baseline _ 
Conflicting — 


Figure 11. Regression Plot for Inter-Rater Reliability Difficulty and Overall Total Time. 


b. Adjoined Times 


For situation time ambiguous/missing information was_ significantly 
greater than conflicting information (p < 0.01), and significantly greater than baseline 
information (p < 0.01). However, there was no significant difference between conflicting 
information and baseline information (p = 0.45). Furthermore, scenario difficulty was a 
significant predictor for ambiguous/missing situation time (p < .01), but not a significant 
predictor for conflicting situation time (p = 0.12) and baseline situation time (p = 0.33) 


(see Figure 12). 
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Figure 12. Regression Plot for Scenario Difficulty for Adjoined Times. 


For additional time, baseline information was significantly less than 
conflicting information (p < 0.01), in addition to, ambiguous/missing information being 
significantly less than conflicting information (p < 0.01). However, there was no 
significant difference between ambiguous/missing information and baseline information 
(p = 0.10). Furthermore, scenario difficulty was a significant predictor for 
ambiguous/missing additional time (p < 0.01) and conflicting additional time (p < 0.01), 


but not a significant predictor for baseline additional time (p = 0.07). 


For time to decision, ambiguous/missing information was significantly 
greater than baseline information (p < 0.01). Conversely, there was no significant 
differences between ambiguous/missing information and conflicting information (p = 
0.10) or conflicting information and baseline information (p = 0.32) (see Figure 13). 


Interestingly, scenario difficulty was not a significant predictor for time to decision. 
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Figure 13. Participant Mean Time to Decision for the Categories of Uncertainty. 


Cc. Interrelated Times 


For situation time/additional time, the researchers found significant 
differences between all three categories of uncertainty: (1) ambiguous/missing 
information was significantly greater than conflicting information (p < 0.01), (2) 
ambiguous/missing information was significantly greater than baseline information (p < 
0.01), and (3) conflicting information was significantly greater than baseline information 
(p < 0.01). For time to decision/additional time, the researchers found two significant 
differences: (1) ambiguous/missing information was significantly greater than baseline 
information (p < 0.01) and (2) conflicting information was significantly greater than 
baseline information (p < 0.01). Additionally, scenario difficulty was a significant 
predictor for both ambiguous/missing information situation time/additional time and 


ambiguous/missing information time to decision/additional time. However, it was not a 
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significant predictor for conflicting information situation time/additional time, conflicting 
information time to decision/additional time, baseline information — situation 


time/additional time, and baseline information time to decision/additional time. 


d. Within Category Times 


For ambiguous/missing information, the researchers found a significant, 
positive correlation between situation time and time to decision (r = 0.54, p < 0.01). 
However, there were no significant correlations between situation time and additional 
time (r = 0.34, p = 0.08) and between additional time and time to decision (7 = 0.22, p = 
0.24). 


For conflicting information, the researchers found two significant 
correlations: (1) situation time and time to decision (r = 0.64, p < 0.01) and (2) additional 
time and time to decision (r = 0.50, p < 0.01). However, no significant correlations were 


found between additional time and situation time (r = 0.26, p = 0.17). 


For baseline information, the researchers found significant, positive 
correlations between all three time segments: (1) situation time and additional time (r = 
0.49, p < 0.01), (2) situation time and time to decision (r = 0.53, p < 0.01), and (3) 


additional time and time to decision (r = 0.51, p < 0.01). 


B. SUMMARY STATISTICS FOR UNCERTAINTY RESPONSE SCALE 


URS data was produced through a scoring system on a Likert Scale, thus non- 
parametric tests (Spearman’s Rho) were used to determine correlation. Additionally, 
descriptive statistics provide means, medians, and standard deviations for each category 
of the URS. For this study, EU served as the primary metric of analysis. CU and DFC 


were recorded for supplementary analysis and future research. 


1. Descriptive Statistics for URS 


CU had the highest mean score of 62 (SD = 8), while EU had the lowest mean 
score of 32 (SD = 7). Additionally, DFC produced a mean score of 59 (SD = 7). The 


median score for CU was 64, while the median score for EU was 33. DFC had a median 
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score of 58. The small standard deviations coupled with the medians being relatively 


close to the means suggest a small amount of variance amongst the URS scores (see 


Table 5). 


| Category | EU | DFC | Cu | 


Standard Error 
Standard Deviation 








Maximum 


Table 5. Descriptive Statistics for URS. 


2: Inferential Statistics for URS 


The researchers found no significant correlations between the categories of the 
URS: (1) CU and EU (p = 0.15, p = 0.44), (2) CU and DFC (p = -0.22, p = 0.25), and (3) 
EU and DFC (p = -0.30, p = 0.12). These findings suggest that the URS measures distinct 


characteristics or variables, and are not associated or related to one another. 


C. SUMMARY STATISTICS FOR EXPERIENCE 


Operational experience was discussed in Chapter II.C.3. However, a more 
detailed analysis is warranted due to the inclusion of experience as a hypothesis. 


Consequently, descriptive statistics will be provided in this section. 


1. Descriptive Statistics for Operational Experience 


The participants had a mean of 12 (SD = 6) months of operational experience in 
Iraq and a mean of 3 (SD = 5) months of experience in Afghanistan. However, the 
median months in Afghanistan are 0, while the median months in Iraq are 12. The highest 
number of months of operational experience in either theatre is 25 months. The variance 


for experience in each theatre can be seen in Figure 14. 


54 


Nineteen of the 28 participants did not serve in Afghanistan. Three participants 
had not served in Iraq, and 6 participants served in both theatres. Overall operational 


experience had a mean of 14 months (SD = 7), and a median of 12.5 months (see Figure 
1X 


25 


oa 





Y 


* ——Time in Afghanistan in months 
—Time in lraq in months 


Figure 14. Participants Months of Operational Experience by Theatre. 





Frequency 





Figure 15. Distribution of Total Operational Experience in Months. 


J 


D. SUMMARY STATISTICS FOR VARIABLE ASSOCIATIONS 


The researchers found no significant correlations between categories of 


uncertainty for overall total time, the categories of the URS, and operational experience. 
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Vv. DISCUSSION 


The discussion is divided into three parts: (a) implications of analysis, (b) 
constraints and limitations, and (c) conclusions and recommendations. Part A will the 
implications of the statistical findings presented in IV. Additionally, it will outline where 
and possibly why those statistical findings emerged. Part B will discuss the constraints 
and limitations of the research based on the data from Chapter II[ and Chapter IV. It will 
discuss important observations obtained from the entire body of the researchers’ work. 
Additionally, it will describe how our research relates to the prior work described in 
Chapter II and Chapter III. Finally, Part C will discuss the short-term and long-term 


vision of research in decision making under uncertainty. 


A. IMPLICATIONS OF ANALYSIS 
1. Hypothesis One 


Ha,: There is a significant difference in the overall total time with respect 
to the three categories of uncertainty. 


The results support the alternative hypothesis that there are significant differences 
between the categories of uncertainty with respect to overall total time. The results 
suggest that decision makers respond slower to situations with ambiguous/missing 
information compared to baseline information or conflicting information. Decision 
makers respond in statistically the same amount of time when faced with conflicting 
information or baseline information. These findings produced a subsequent set of 


questions concerning the nature of these differences. 


The differences between ambiguous/missing information and _ baseline 
information were found in situation time, time to decision/additional time, time to 
decision, and scenario difficulty. The results suggest that decision makers spent more 
time in the situation time and time to decision segments for ambiguous/missing 
information compared to baseline information. As expected, ambiguous/missing situation 
time and ambiguous/missing time to decision were positively correlated. Thus, if a 


decision maker spent a large amount of time on situation time, then he/she would spend a 


ay, 


large amount of time on time to decision, and visa versa (see Figure 16). Finally, the 
results suggest that scenario difficulty was a determining factor in response time (i.e., the 
higher the level of difficulty, the slower the individual responded to ambiguous/missing 
scenarios). The level of difficulty was not a significant factor for baseline overall total 


time. 


Ambiguous-Missing Information > Baseline Information 





Ambiguous-Missing ST > Baseline ST Ambiguous-Missing TTD > Baseline TTD 











Ambiguous-Missing AT = Baseline AT 


Supplementary Note 
Ambiguous-Missing TTDAT > Baseline TTDAT 


Ambiguous-Missing ST is positively correlated to Ambiguous-Missing TTD 





Figure 16. Diagram of Statistical Inferences for Differences between 
Ambiguous/Missing Information and Baseline Information. 


The differences between ambiguous/missing information and _ conflicting 
information for decision time and scenario difficulty are additional time and situation 
time. The results suggest that decision makers spend more time on ambiguous/missing 
additional time compared to conflicting additional time. Additionally, the results suggest 


that decision makers spend more time on ambiguous/missing situation time compared to 
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conflicting situation time. Furthermore, both conflicting situation time and additional 
time were positively correlated to time to decision. Thus, if a decision maker spends a 
large amount of time on either conflicting situation time or additional time, he/she would 
spend a large amount of time on time to decision. And as in the previous paragraph, 
scenario difficulty was a predictor for ambiguous/missing overall total time, but not a 
predictor for conflicting overall total time (i.e., scenario difficulty had no effect on 
conflicting overall total time, but did have an effect on ambiguous/missing overall total 


time). 


Finally, the results suggest that if a decision maker is slow to respond on any of 
the three categories of uncertainty, then he/she will be slow on the others. Thus, the 
researchers can infer that the response time to any of the categories of uncertainty is 


correlated with a decision maker’s response time to others. 


a. Ambiguous/Missing Overall Total Time and Baseline Overall 
Total Time 


While the four preceding paragraphs describe “where” the differences lie, 
an explanation of “why” is needed. To address the differences in ambiguous/missing 
overall total time and baseline overall total time, the researchers offer three possibilities: 
(1) the differences are a result of the relationship between uncertainty and difficulty, (2) 
the ambiguous/missing information presented in the additional information (AI) did not 
have any effect because the decision maker had thoroughly evaluated the situation 
information (SI) and produced multiple COAs, and (3) the ambiguous/missing 
information in the AI acted as a retarding agent when merged with the situation 


information (SI). 


As noted, the researchers attempted to evenly distribute the information in 
both the additional time and situation time sections. Thus, one portion of the slide would 
not possess more or less information than another section of the slide. The researchers did 
conduct reliability analysis to determine the difficulty of each scenario (see Chapter 
I1.B.4). They did not conduct reliability analysis on the complexity. This is where the 


discussion about the relationship between complexity and difficulty becomes relevant. 
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Because our reliability inquired about the difficulty of the situation, and the not 
complexity, these results can only address the former. With that said, difficulty is created 


by numerous factors, including context, uncertainty, task, and mission type. 


It is not clear whether uncertainty produces difficulty or difficulty 
produces uncertainty. As previously stated, conflicting information emerges when two or 
more sources or information contradict one another. Conversely, ambiguous/missing 
information can be available or unavailable to the decision maker. If available it may be 
unclear to its meaning or relevance to the mission; if unavailable the decision maker is 


unable to access it. 


To create an ambiguous/missing situation, the researchers had to draft a 
scenario with detailed information or an abundance of resources, which produced 
numerous alternatives for the decision maker to compare and contrast. The decision 
maker may spend more time reasoning about the situation, comprehending the facts, and 
gaining SA (hence, the difficulty previously noted). In terms of this study, SI was clear 
and concise, providing a context in which actors were defined and environments 
understood. The AI provided the uncertainty. Thus, the main contributor to the 
differences in situational time, time to decision, and overall total time could be the nature 


of the AI. 


A second possibility is that if a decision maker spent a large amount of 
time reasoning about the SI (understanding his overarching mission and how it relates to 
the situation), then he/she may have spent a large amount of time producing a COA, 
regardless of the AI, in the time to decision segment. This may account for why 
additional time was not statistically significant between ambiguous/missing information 
and baseline information (i.e., the decision maker had already begun reasoning about the 
situation information). Thus, the decision maker had already developed COAs, and the 
ambiguous/missing information presented reinforced the rationale of at least one of the 
COAs, or possibly, the uncertainty had little or no impact on his decision-making 


process. 
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This argument has implications for Klein’s (1993) RPD model, where a 
decision maker relies on past experience when choosing a COA. Because our sample 
contained individuals who had operational experience in Iraq, Afghanistan, or both, they 
may have experienced similar situations and missions. Thus, they executed what they 


knew. 


A third possibility arises when SI and AI are combined. Because the 
ambiguous/missing information was not presented until the decision maker requested the 
AI, and SI remained present on the slide, the AI drastically impacted time to decision. 
This is supported by the finding that ambiguous/missing time to decision/additional time 
is significantly greater than baseline time to decision/additional time (p < 0.01), and there 
was no significant difference between ambiguous/missing additional time and baseline 
additional time. The findings suggest that the ambiguous/missing AI acted as a much 
more effective retarding agent when merged with the SI compared to baseline 
information. That is, even if the situation time between ambiguous/missing and baseline 
were equal, the ambiguous/missing information in the AI would slow the decision 
maker’s response when compared to baseline information. More importantly, scenario 
difficulty was a predictor variable in ambiguous/missing situation time. This suggests that 
the more difficult the scenario, the longer time the decision maker’s situation time. This 
is not true for conflicting situation time or baseline situation time. Consequently, the 
researchers believe the decision maker had to re-evaluate all their previous assumptions 
and beliefs, and produce new COAs or iterate previous plans for ambiguous/missing 


scenarios. 


The third argument has implications for the DMSC. As _ previously 
discussed, Shattuck and Miller (2006) stress the importance of promulgation of 
information and feedback loops in DMSC. Consequently, the emergence of 
ambiguous/missing information in AI may alter the lens of the decision maker. Thus, 
his/her interpretation of ground truth is no longer valid, and the decision maker must 


recalibrate his/her lens to achieve effective SA. In many ways, the third argument lends 
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itself to the cliché, “the calm before the storm,” (i.e., the commander assumes his lens is 
properly calibrated to the environment only to be rattled by the emergence of uncertainty 


in the tactical battle space). 


b. Ambiguous/Missing Overall Total Time and Conflicting Overall 
Total Time 


While the differences described above are difficult to express, the 
disparity in ambiguous/missing overall total time and conflicting overall total time is 
even more muddled. The most straightforward way to discuss the differences between 
ambiguous/missing overall total time and conflicting overall total time was to examine 
why conflicting overall total time was significantly less than ambiguous/missing overall 


total time. 


Once again, the idea of difficulty arises when discussing the differences in 
additional time. It appears that, situation time is significantly different between the 
categories mostly due to the predictor variable of difficulty. While this reason may seem 
somewhat simplistic, it follows from the data analysis reported in the previous chapter 


(see Figure 17). 
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Ambiguous-Missing Information > Conflicting Information 





Ambiguous-Missing ST > Conflicting ST Ambiguous-Missing TTD > Conflicting TTD 











Ambiguous-Missing AT = Conflicting AT 


Supplementary Note 


Ambiguous-Missing TTDAT > Conflicting TTDAT 


Figure 17. Diagram of Statistical Inferences for Differences between 
Ambiguous/Missing Information and Conflicting Information. 


Scenario difficulty was a predictor for both conflicting additional time and 
ambiguous/missing additional time. At a qualitative level, the researchers noticed the 
decision makers re-reading the lines due to confusion or disbelief (i.e., the contradictory 
information prompted the decision maker to ensure the knowledge previously acquired 
was accurate). The decision maker spent a great deal of time re-reading and reinterpreting 


his previous beliefs. 


Unfortunately, this does not adequately explain why conflicting situation 
time and additional time were significantly, positively correlated to conflicting time to 


decision. The results suggest that because additional time and situation time were not 
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significantly correlated, there was no association between the two variables (i.e., just 
because a decision maker was fast in situation time, does not imply he/she was fast in 


additional time). 


The researchers support the findings concerning time to 
decision/additional time. Unlike ambiguous/missing time to decision/additional time and 
baseline time to decision/additional time where the results were not significantly 
different, the findings indicate that time to  decision/additional time for 
ambiguous/missing information was significantly greater than time to decision/additional 
time for conflicting information. Contrary to the third possibility discussed in Chapter 
V.A.1., the researchers suggest that the true difference lies in the SI (i.e., the uncertainty 
introduced in AI retarded both situations equally, thus the time differential is created 
from the aforementioned difficulty element). The significant difference for additional 
time between the categories is diminished by the time to decision between the categories 
(i.e., the time differential that favored conflicting additional time was mitigated by the 
time differential favoring ambiguous/missing additional time). This is further supported 
by the times to decision for ambiguous/missing information and conflicting information. 
Consequently, the researchers suggest that situation information and scenario difficulty 
were the determining factors on whether or not the decision maker responded faster to 


ambiguous/missing or conflicting information. 


Cc. Conflicting Overall Total Time and Baseline Overall Total Time 


In this section the researchers explain why conflicting overall total time 
and baseline overall total time are not statistically different. Situation time between the 
two categories is statistically different, but additional time was not. Moreover, time to 
decision between the categories was not significant, but conflicting time to 
decision/additional time was significantly greater than baseline time to 


decision/additional time. 


While examining the differences between the categories of uncertainty is 
necessary, it is also important to analyze where they are similar. For example, there was 


no statistical difference between conflicting situation time and baseline situation time. 
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Additionally, scenario difficulty was not a predictor for overall total time for either 
category. These two findings are very important. The researchers can suggest that 
uncertainty alone does not determine the rate of response, but rather uncertainty must be 


placed in the context of the situation. 


The differential between baseline additional time and _ conflicting 
additional time can be explained in a manner similar to ambiguous/missing information 
and conflicting information. When the decision maker is presented with contradictory 
information, the researchers often observed many participants re-reading the given 
information to reinforce or reconsider their previous assumptions. This would account for 


the statistical significance across the categories of uncertainty found in additional time. 


By definition, ambiguous/missing information presents the decision maker 
with more alternatives to consider compared to conflicting. However, an argument can 
also be made that baseline information is not much different than conflicting information 
(i.e., having an explicit understanding of the information may be the same as having two 
explicit understandings). Consequently, if a decision maker believes both to be equally 
valid interpretations, then either interpretation is adequate and could be assumed to be 
true. This may explain why there is no statistical difference for time to decision when 
comparing baseline and conflicting information. Moreover, the researchers suggest that 
the time to decision/additional time difference found is not large enough to account for 
the non-significant situation time difference. Thus, time differentials in additional time 
were mitigated by situation time and time to decision. Interestingly, the findings in this 
section support the findings of the third pilot study (see Chapter III.A.3). 

2 2"* Hypothesis 
Haz,: There is a significant, positive correlation between emotional 
uncertainty and overall total time for ambiguous/missing information and 
conflicting information. 
Haz: There is a significant, negative correlation between cognitive 


uncertainty and overall total time for ambiguous/missing information and 
conflicting information. 
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The results refute the alternative hypothesis that cognitive uncertainty is 
significantly, negatively correlated with ambiguous/missing overall total time or 
conflicting overall total time. Additionally, the results refute the alternative hypothesis 
that emotional uncertainty is significantly, positively correlated with ambiguous/missing 


overall total time or conflicting overall total time. 


a. Cognitive Uncertainty and Overall Total Time 


The findings for cognitive uncertainty are interesting. The characteristics 
of environments in which naturalistic decision making occurs (Orasanu and Connolly, 
1993) are prevalent in the military. However, this does not imply that structure does not 
exist. Instead, the researchers suggest that at the beginning of most military operations 
there is structure, but uncertainty can still exist in the battle space. Consequently, it is not 


uncommon for structure and uncertainty to coexist for the decision maker. 


This relationship leads to anomalies when examining cognitive uncertainty 
and response time. Because the military places a great deal of emphasis on C2, 
commanders will strive to avoid organizational chaos. They will more than likely have an 
organizational structure that is accustomed to or designed to mitigate uncertainty. This 
may explain why cognitive uncertainty had the largest average score compared to the 
other categories of the URS. It may well be that the participants were accustomed to 
operating within a structured environment. Thus, the presence of uncertainty is not 
unexpected or foregone to their cognitive performance. Instead, it is assumed to be a part 


of everyday operation. 


b. Emotional Uncertainty and Overall Total Time 


The lack of statistical significance for emotional uncertainty and overall 
total time can be explained in a manner similar to the previous discussions. However, it is 
important to note some characteristics of the emotional uncertainty data prior to 
proceeding to the aforementioned case. Emotional uncertainty had the smallest range (R 


= 29) and variance (v = 50) compared to the other URS categories. This was expected 
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because the sample pool consisted of military officers who often operate in dynamic, 
uncertain environments. Regardless, the researchers were somewhat surprised by the 


findings. 


Maladaptive behavior toward uncertainty does not necessarily lead to a 
failure to respond quickly. Although a decision maker may experience anxiety or sadness 
when faced with uncertain conditions, he/she may still be able to make a quick decision 
due to other factors overriding their emotional response. Future research must be 
conducted to better understand why emotional uncertainty does not directly affect the 


response time of the decision maker. 


3. 3"! Hypothesis 


Ha3: There is a significant, negative correlation between operational 
experience and overall total time for ambiguous/missing information and 
conflicting information. 


The results refute the alternative hypothesis that operational experience is 
negatively correlated with ambiguous/missing overall total time and conflicting overall 
total time. This research does not support the research conducted by St. John et al. 
(2000). However, it is important to note St. John et al. (2000) used senior and junior 
ranks to conduct their study, and not years of service. However, their overarching 


assumption was that higher ranks equal more experience. 


Once again, the researchers were surprised by this finding. The absence of a 
relationship suggests that operational experience does not increase the speed of battle 
under uncertainty (i.e., more operationally experienced decision makers do not make 
faster decisions under uncertain conditions, nor as the data suggest, do they make slower 


decisions). 


Other than one participant, all the participants had at least six months of 
operational experience. The researchers omitted all participants without any operational 
experience. Consequently, the participants operated in Afghanistan, Iraq, or both. The 
data did not provide the researchers with the ability to discriminate performance based on 


level of experience. 
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B. CONSTRAINTS AND LIMITATIONS OF THE RESEARCH 


The researchers experienced two challenges during the study. First, defining the 
relationship or association between complexity and difficulty is necessary to 
understanding uncertainty. Second, branch participant selection is important due to the 
tasks assigned by the scenario. As discussed previously, the relationship between 
complexity and difficulty is important in terms of their influence on uncertainty. The 
researchers’ reliability survey inquired only about difficulty and not complexity. These 
terms are most likely related but not necessarily synonymous. Scenario difficultly has 
contributed greatly to our work by influencing decision time across the categories of 


uncertainty. 


While the relationship of complexity and difficulty is important, participant 
selection based on combat branch is equally vital to achieving better results. While 
participants were procured from all three branch categories of the U.S. Army, the 
scenarios were constructed from a CA perspective (i.e., the situations resembled the types 
of tasks and mission executed by infantry, armor, and artillery officers). This is not to say 
that the CS officers or CSS officers cannot accomplish the missions evaluated by the 
study. In fact, the combination of smaller, lighter forces along with the tactics of post 
September 11, 2001 insurgents have placed many CS and CSS officers in direct contact 
with the enemy. Our suggestion points out that the branches associated with CS and CSS 
receive different training and, for the most part, do not perform enemy engagement 
missions as frequently as their CA counterparts. Consequently, the exclusion of CS and 


CSS officers would bring greater validity to the results. 


Cc. CONCLUSIONS AND RECOMMENDATIONS 


The conclusion and recommendations section contains three parts: (1) 
conclusions, (2) future research, and (3) summary. The conclusion will discuss the 
important information ascertained from the study, while future research will offer 
suggestions for future work in the field of uncertainty. Finally, the summary discusses the 


researchers’ final thoughts on the study. 
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1. Conclusions 


First, the evidence suggests that decision makers respond differently when 
confronted with various categories of uncertainty. Additionally, scenario difficulty was a 
significant factor in ambiguous/missing information response time, but not a significant 
factor in determining conflicting or baseline information response time. These findings 
suggest that the categories of uncertainty coupled with scenario difficulty determine 
response time. However, whether or not these differences are mostly due to the categories 
of uncertainty, scenario difficulty, or both could not be determined by this study. 
Consequently, future research must focus on teasing out these differences to determine 


what role each one of these factors plays in response time. 


Second, the absence of an association between operational experience and 
response time is dubious. However, this study did not look at the quality of decisions, but 
focused on response time. The true differences may be found in the quality or type of 
decision vice the response time. This theory is being explored by TRAC-FLVN in an 
adjacent study. Third, because there was no association between the analyzed categories 
of the URS and overall total time, the researchers believe calibration may have been the 
cause (i.e., because military officers operate in uncertain environments, they have learned 
how to mitigate its effects to remain efficient). However, future research is needed to 


determine if this theory is correct. 


2. Future Research 


The conclusions provide a wealth of questions for future research. These 
questions can be grouped into four areas of study: (a) additional research on decision- 
making response to uncertainty, (b) application to modeling and simulation for human 
behavior in software agents, (c) expansion to military services, and (d) application to 


other fields of occupation. 


a. Additional Research on_ Decision-Making Response _ to 
Uncertainty 


This study explores the categories of uncertainty in terms of response 


time. However, response time is just one of many performance measures that can be 
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examined to determine how individuals respond under uncertainty in tactical situations. 
TRAC-FLVN will be publishing an adjacent report on this study that classifies the 
decisions made by the participants in terms of passive, cautious, and decisive action. This 
analysis will help determine what types of decisions individuals make when faced with 
the different categories of uncertainty. Additionally, research is needed to explore the 
relationship between complexity and difficulty and their impact on uncertainty. By doing 
so, researchers will be able to develop methods that can tease out the influence of these 


constructs and provide more conclusive findings. 


b. Application to Modeling and Simulation for Human Behavior 


This study provides the foundation for creating software agents that better 
model individual behavior under conditions of uncertainty. Modeling human behavior is 
a difficult task, but it is vital to understanding decisions made at the tactical level. While 
various types of uncertainty lead to differences in response time, research must focus on 
other factors that may also affect decision makers. The injection of harsh weather or 
unbearable heat may drastically affect an individual’s ability to reason under uncertainty. 
Acute of chronic fatigue may also affect a decision maker’s ability to reason about 
uncertainty. Data that result from the interactions of these factors may produce more 


robust algorithms to better model human behavior. 


c. Expansion to the Military Services 


The scenarios used in this study were crafted for U.S. Army personnel. 
However, uncertainty is not only found in U.S. Army operations, but in other military 
services as well. Scenario generation should be developed in other tactical environments, 
including continuous operations at sea and air defense operations at home and abroad. 
For example, U.S. Navy carrier strike groups’ operations routinely deal with uncertainty 
when coordinating organic and non-organic assets. Additionally, combat operations, and 
in particular, amphibious operations, conducted by the U.S. Marine Corps parallel the 
tactical and operational environments experienced by those in the U.S. Army. The 


challenge is to develop scenarios that provide tactical realism and meet the specific 
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human behavior modeling needs of the services. An even more challenging approach 
would be to inject categorical uncertainty into staff operations to determine how it affects 


team, staff planning operations, and both at the single service and joint levels. 


d. Application to Civilian Occupations 


While modeling military operations and decision makers was the central 
focus of this research, examining the impact of uncertainty on occupations outside the 
military may also prove beneficial. Businesses routinely operate with uncertainty at all 
levels. For example, commodity speculators attempt to determine the future price of oil 
based on complex, dynamic demand and supply issues in the near and long term. 
Additionally, transportation and logistics companies face uncertain conditions in weather, 
prices, and interruptions in supply lines when making critical operational decisions that 
have long term consequences. Developing software agents that mimic human behavior 
may help us understand these uncertainties by properly modeling how they influence 
human decision making. Such research could provide additional insights into modeling 
and simulation techniques and how decision makers outside the military cope with 


uncertain conditions. 


3. Summary 


Research in decision making under uncertainty has great potential. The study of 
human reasoning under uncertain conditions is critical to understanding the dynamic and 
unpredictable nature of war. The researchers are confident that the findings in this study 
will add to the body of knowledge in this field of decision making under uncertainty and 
lead to improved comprehension of the battle space. Additionally, these proposed 
research areas will provide a more robust understanding of uncertainty in modeling and 
simulation. The results of the research coupled with the improved modeling and 
simulation will allow commanders to eventually view the battle before it is fought. These 
improvements have the potential to revolutionize the manner in which military leaders 


train for, reason about, and conduct military operations at all levels of war. 
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APPENDIX A. UNCERTAINTY RESPONSE SCALE TEST 


. [tend to give up easily when I don’t clearly understand a situation. 
4 
Never Sometimes Now and Often Always 
Then 


2. When I go shopping, I like to have a list exactly of what I need. 


2 3 
Never Sometimes Now and Often Always 
Then 


3. I feel better about myself when I know that I have done all I can to accurately plan my future. 
1 4 


Never Sometimes Now and Often Always 
Then 


4. Sudden changes make me feel upset. 
] 4 
Never Sometimes Now and Often Always 
Then 
5. When making a decision, I am deterred by the fear of making a mistake. 
1 3 
Never Sometimes Now and Often Always 
Then 
6. When uncertain, [ act very cautiously until I have more information about the situation. 
I 2 4 5 


Never Sometimes Now and Often Always 
Then 
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7. [like to have things under control. 
l 3 


Never Sometimes Now and Often Always 
Then 


8. When the future is uncertain, I generally expect the worst to happen. 
1 4 
Never Sometimes Now and Often Always 
Then 


9, Facing uncertainty is a nerve-wracking experience. 


Never Sometimes Now and Often Always 


10. I get worried when a situation is uncertain. 
] 
Never Sometimes Now and Often Always 
Then 
11, Thinking about uncertainty makes me feel depressed. 
l 
Never Sometimes Now and Often Always 
Then 


12. | find the prospect of change exciting and stimulating. 


Never Sometimes Now and Often Always 


Then 
13, Uncertainty frightens me. 
Never Sometimes Now and Often Always 
Then 
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14. There is something exciting about being kept in suspense. 
4 
Never Sometimes Now and Often Always 
Then 


15. The idea of taking a trip to a new country fascinates me. 
2 4 


Never Sometimes Now and Often Always 
Then 


16. I like going on holidays with nothing planned in advance. 
Never Sometimes Now and Often Always 
Then 
17. I think you have to be flexible to work effectively. 


Never Sometimes Now and Often Always 
Then 


18, Taking chances is part of life. 
Never Sometimes Now and Often Always 
Then 
19. When I feel uncertain about something, I try to rationally weigh up all the information I have. 


4 
Never Sometimes Now and Often Always 
Then 


20. Before making any changes, | need to think things over thoroughly. 


4 
Never Sometimes Now and Often Always 
Then 


oes 


21. I prefer to stick to tried and tested ways of doing things. 


2 4 
Never Sometimes Now and Often 
Then 
22. | like to have my weekends planned in advance. 


1 4 
Never Sometimes Now and Often 
Then 


23. I feel curious about new experiences. 


4 


Never Sometimes Now and Often 
Then 


24. I like to think of a new experience in terms of a challenge. 


1 4 
Never Sometimes Now and Often 
Then 


25. A new experience is an occasion to learn something new. 


2 
Never Sometimes Now and Often 
Then 


Always 


5 


Always 


Always 


Always 


Always 


26. When | feel a situation is unclear, I try to do my best to resolve it. 


Never Sometimes Now and Often 
Then 


27. 1 like to know exactly what I’m going to do next. 


3 
Never Sometimes Now and Often 
Then 
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5 
Always 


5 
Always 


28. When facing an uncertain situation, I tend to prepare as much as possible, and then hope for 
the best. 


3 5 
Never Sometimes Now and Often Always 
Then 


29. I feel relieved when an ambiguous situation suddenly becomes clear. 
l 
Never Sometimes Now and Often Always 
Then 


30, When I feel uncertain, I try to take decisive steps to clarify the situation. 


Often Always 


Never Sometimes Now and 
Then 


31. When I can’t clearly discern situations, | get apprehensive. 
Never Sometimes Now and Often Always 
32. | enjoy finding new ways of working out problems. 
4 
Never Sometimes Now and Often Always 


Then 


33. When I'm not certain about someone's intentions towards me, I often become upset or angry. 


| 2 
Never Sometimes Now and Often Always 
Then 
34. New experiences can be useful. 
] 2 5 
Never Sometimes Now and Often Always 
Then 


TT. 


35. When uncertain about what to do next, I tend to feel lost. 
1 2 4 
Never Sometimes Now and Often Always 
Then 


36. | feel anxious when things are changing. 


3 4 5 
Never Sometimes Now and Often Always 
Then 
37. New experiences excite me, 
1 2 3 4 
Never Sometimes Now and Often Always 
Then 
38. | think variety is the spice of life. 
4 
Never Sometimes Now and Often Always 
Then 


39. L try to have my life and career clearly mapped out. 


4 5 
Never Sometimes Now and Often Always 
Then 


40. | think a mid-life career change is an exciting idea. 
4 
Never Sometimes Now and Often Always 
Then 


4]. When a situation is unclear, it makes me feel angry. 


l 2 3 
Never Sometimes Now and Often Always 
Then 
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42. | enjoy unexpected events. 


y 3 + 
Never Sometimes Now and Often Always 
Then 


43, I like things to be ordered and in place, both at work and at home. 


3 4 
Never Sometimes Now and Often Always 
Then 


44. I really get anxious if I don’t know what someone thinks about me. 


l 2 4 
Never Sometimes Now and Often Always 
Then 


45, I easily adapt to novelty. 
l 


Never Sometimes Now and Often Always 
Then 


46. | am hesitant when it comes to making changes. 
1 2 4 
Never Sometimes Now and Often Always 
Then 
47. I like to plan ahead in detail rather than leaving things to chance. 
1 4 5 
Never Sometimes Now and Often Always 
Then 
48. Before | buy something, I have to view every sample [ can find, 
2 4 


Never Sometimes Now and Often Always 
Then 
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APPENDIX B. MODIFIED CDM INTERVIEW SHEET 





Recall a tactical situation during a recent deployment to a combat zone in which you were confronted with 
uncertainty and had to make a decision. Uncertainty can be described simply as a sense of doubt that blocks or 
delays action. There are no right or wrong answers and we are in no way evaluating your performance or the 


Critical incident summary 
(Location, Date/Time, Key 
Personnel, General Description) 


Detailed Incident timeline 
(Sequence of key incidents, 
activities) 


Identify uncertainty (uncertainties) 
in the timeline 





Describe the specific nature of the 
uncertainty (what was known by 
whom, what was not known by 
whom, what the participant 
wanted/needed to know in that 
context) 


Describe the actions that were 
undertaken to overcome the 
uncertainty (e.g., asked for 
additional data or Intel, relied on 
previous experiences, etc.) 


Describe the effect of the 
uncertainty on the decision that 
was made 





Describe what you would do 
differently if faced with the same 
situation 


Table 6. © CDM Uncertainty Elicitation Interview Sheet. 
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APPENDIX C. NPS PILOT STUDY # 1 SCREEN SHOTS 


TARGET SET 





PROCEED 





WORD 18 of 30 a 


Figure 18. Low Resolution Ambiguity Slide. 


TARGET SET 





tac tan tab 


tag tam tap 


PROCEED 


WORD 19 of 30 p> | 





Figure 19. Medium Resolution Ambiguity Slide. 
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TARGET SET 





rake rate rare 


raze face fave 


PROCEED 








WORD 17 of 30 [> 


Figure 20. High Resolution Ambiguity Slide. 
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APPENDIX D. NPS PILOT STUDY #2 SCREEN SHOTS 





Figure 21. Conflicting Information Slide. 





Figure 22. Missing Information Slide. 
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Figure 23. Ambiguous Information Slide. 





Figure 24. Baseline Information Slide. 
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APPENDIX E. CGSC TACTICAL VIGNETTES 





Situation 
n| 3. Sima a. You are a MITT team leader advising an 
@ sods (at Report) Iraqi BN. 
| b. Insurgents have been operating in the 
\ area to include emplacement of IEDs 
a7 and ambush sites. 
\ c. The MSR has been swept for IEDs 
SS eres (und topo) within the last (2) hours and there is no 
Rey ) evidence of IEDs or ambush. 
SEE Poa Additional Information 


1. Near dark, the Iraqi BN Commander receives a report of (6) dead bodies 
along the MSR at PointA. 

2. You accompany the Iraqi CDR and a heavily armed Iraqi Police (IP) escort 
south along the ASR to Point A. 

3. While enroute, in the vicinity of CKP 1, you receive a radio report directly 
from your higher HQ that the dead bodies are located on the MSR at Point 
B, to your rear, in a location you have already driven past. 


PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 
SLIDE 1 


Figure 25. Conflicting Information Slide. 


Situation 


a. You are a Special Forces ODA team leader 
operating with (50) freedom fighters. Your 
splitODA teams (mounted) are occupying the 

% Hes Os > (2) ABF positions. 

S po b. Your mission is a coordinated daylight attack 
against the remnants of a light infantry BN 
(minus) with approximately (200) entrenched 





ona hillside. 
c. The enemy BN has mortars and small arms 
capability. 
Town d. Prior to the attack, an airstrike will reduce 


the enemy BN strength to an acceptable 
attac k/defend ratio. 


Additional Information 


1. You observe several rounds ‘off target” during the air strike. 
2. The adjacent units report they started their attacks. 
3. You do not have any information about the remaining enemy strength. 


PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 
SLIDE 2 


Figure 26. Ambiguous/Missing Information Slide. 
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Situation 


a. You are leading an infantry platoon raid to 
detain a suspected anti-coalition leader 
tunning a weapon's black market in OBJ 
Able. 

b. A platoon blocking force is moving north on 
Route B lack, and will link uo with your 
assault force to secure the objective. 

c. The two platoons have enough combat 
power to secure allroutes leading out of the 
obje ctive. 





Additional Information 


. The blocking force reports being set in position to the South and all escape 
routes are secured. 

. You continue to OBJ Able, enter crowded market conditions, and observe 
the anti-coalition leader in a dark blue BMW sedan. 

. The dark blue BMW sedan is unable to move due to congestionin the 
market area. 


PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 
SLIDE 3 


Figure 27. Baseline Information Slide. 


sins Situation 
a. You are aCO CDR incharge of a remote 
FOB. 
b. You have a three vehicle HMMWV QRF 
= available. 
“ros “Sy c. There are (2) NAls within a 5 KM range that 
show evidence of recent occupation and 


discarded military equip ment: (1) a hilltop to 
the North (NAI 1) with some hardened 
NAI2 Be buildings, and (2) an abandoned airport to the 
South (NAI 2). Both sites have been used to 
launch mortar attacks against the FOB. 
d. The CO usually responds to indirect fire by 
locating and destroying the mortar sites. 


Additional Information 


At0105 Hrs, you are locatedin your CO TOC, and hear the sound of 
incoming indirect fire. The first round lands within 50 meters ofthe TOC. 
Your 1SG tells you thata PSG and himself believe the round came from 
the South. 

. Your XO and a guard on the roof report they believe the round came from 
the North. 

PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 

SLIDE 4 


Figure 28. Conflicting Information Slide. 
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Suspicious. 


Situation 


a. Youare the Night OIC responsible for the Division 
ASP security. 
b. ASP stocks are stored on the ground on unimproved 











Movement Avenger 
Xe surfaces and cover a 2km x 2km area. 
~ NK. c. (3) Avengers (ADA Gun) with FLIR capability are 
PA : 7 manned to support security operations. 
/ 7 %. d. A dismounted Q RF with 6 personnel and equipped 
ASP es > with small arms and night vision devices is co- 
LJ - Bil located at the CP. 
ildngs 
e. SOP states the QRF should notbe deployed until 
: enemy activity is confirmed. 
I 
0 1 2 KM 


Additional Information 


It is approximately (5) minutes after shift change. The security personnel 
were just replaced and should be heading back to their respective platoon 
CPs. 
A Specialist from the QRF reports movement near the small arms 
munitions stocks. The Avenger Crews report they cannot observe 
aforem entioned location. 
Previously, locals have looted the stocks and sold ammunition on the 
black market. 

PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 

SLIDE 5 


Figure 29. Ambiguous/Missing Information Slide. 


Situation 


a. You are the OIC responsible for convoy 
security. 
b. During convoy operations, a local 
5 & Convoy Fatality national is killed in a fatal car accident 
a involving one ofthe military convoy 
% vehicles near CKP 2. 
c. Battalion SOP and TTP is to remove 
0 military and civilian bodies to a Mortuary 
\ Affairs team co-located at an Iraqi Police 
° a” Cg Station regardless of political or family 
Mortuary affiliation. 


& 


Additional Information 


You secure the area and report the accident and fatality to higher HQ. 
You identify the body as the cousin of a local leader. 

You have sufficient com bat power on hand to both secure the convoy and 
escort the body to the Mortuary Affairs team. 

The Mortuary Affairs team is located approximately 20 KM south of the 
accident site. 


PLEASE SNAP YOUR CLIC KER WHEN YOU HAVE MADE A DECISION 


SLIDE 6 


Figure 30. Baseline Information Slide. 
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Situation 
a. You are the QRF platoon leader 
responding to a report of dead bodies 
along the MSR. 
b. Insurgents have been operating in the 
area to include emplacement of IEDs 
and ambush sites. 





\ . c. The local townspeople are sympathetic 
ar ) to the insurgents. 
0123 4 5 KM d. Itis not safe to bein this area after 
dark. 


Additional Information 


1. You find (5) dead bodies just before dark. They were killed execution style. 

2. A local reports that several men in a truck dumped the bodies approximately 
(1) hour earlier and then continued South. 

3. A different local indicates the bodies were executed at the site by insurgents 
less than (30) minutes ago. The local gives you an exact location where the 
insurgents went inside the town. 


PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 
SLIDE 7 


Figure 31. Conflicting Information Slide. 














[anche Site Situation 
Scout Z . . 
Po ® a. YouareaCO CDR conducting nightly ambush 
\ aes 2 mis sions outside the FOB to destroy enemy rocket 
a wh F launcher teams. 
See wl al vw b. Your CP is located inside the nearby town. You 
a ee oe 4 have (3) platoons available; 3R° Platoon is the main 
Or effort; the Battalion Scout platoon is occupying the 
: SBF position. 
Suni c. CAS (F-16’s) is available and a TACP is attached; 
Apache Helicopters are in direct supportof the 
" unit’s operations. 


d. You are atthe limit of yourcom munications ranges 
to the platoons and to higher BN he adquarters. 


Additional Information 


1. You receive a relayed report that the 37° platoon leader has been trying to 
reach you. 3"9 platoon is reporting contact with dismounts N orthwest of 
their location. 

2. All units have lost radio com munications with the Scout Platoon manning 
the SBF position. Several attempts to raise them on the radio have failed. 

3. From the relayed information and the locations on the FBCB2, you are not 
sure ifthe platoon leader is actually under fire from insurgents or under 
attack by the friendly SBF position. 

PLEASE SNAP YOUR CLIC KER WHEN YOU HAVE MADE A DECISION 
SLIDE 8 


Figure 32. Ambiguous/Missing Information Slide. 
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Susficious a 
Activity 


\~ Guard 
Sower#2 gt b 
ee 
> 
\ c 
-——_ 
KM 1 
d. 


Situation 


. You are the Night Battle Captain in the 


BOC and are responsible for FOB 
security. 


. Your BN is the only unit operating from 


this FOB. 


. The BN QRF consists of (4) Bradley 


Fighting Vehicles with (4) dism ounts 
each. 

As part of your duties, you have been 
given authority to exercise the QRF in 
support of FOB security. 


Additional Information 


. Guard Tower #1 reports suspicious enemy activity (lights/movem ent) 1 KM 
Northwest of the FOB in the vicinity of the road. 

. Guard Tower #2 confirms observing (2) civilian pickup trucks, and reports 
observing enemy small arms tracer fire incoming from that area. 

. A guidons call and a status check confirms thatno units are currently 


operating outside of the FOB. 


PLEASE SNAP YOUR CLICKER WHEN YOU HAVE MADE A DECISION 
SLIDE 9 


Figure 33. 
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Baseline Information Slide. 
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APPENDIX F. CGSC STUDY INSTRUCTIONS 


Operations Research Department Lawrence Shat tuck, Ph.D. 
Hum an Systems Integration Igshattu @nps.edu 
Glasgow Hall 


Naval Postgraduate School 
Monterey, California 93943 





From: LTJG K.E. Kemmerer, USN 
To: CGSC Decision-Making Participants 


Sbj: DECISION-MAKING STUDY INSTRUCTIONS 
***PLE ASE READ THE FOLLOWING INSTRUCTIONS ALOUD *** 


1. You are about to participate in adecision making study. Please read these directions 
care fully and aloud. We thank you in advance for your time and involvement. 

2. This study consists of several different, independent scenarios. Each scenario 
consists of two blocks of information: (1) situational data and (2) additional data. 
You will read both the situational and additional data aloud. 

3. The situational information provides data conceming the tactical environment, force 
positioning, and the location of the scenario. After you read the situational data 
and are ready to read the additional data, snap the clicker provided to you. 
The additional information will provide real-time data describing the events and 
intelligence occurring on the battlefield. 

4. After you have read both blocks of data out aloud, you will be asked to make a 
decision. Please indicate you have made a decision by snapping your clicker. 

5. Once you snap your clicker, you will be asked to answer two questions: 

1) To provide a brief description of your decision? 

2) How did you arrive at your decision? 
Please do not snap your clicker until you are ready to answer these two 
questions and speak clearly into the microphone while doing so. 

6. After you have answered both questions, snap your clicker to proceed to the next 
scenario. 


7. Please test that your clicker is working properly now. 


Once again, thank you for your time, and please answer to the best of your ability. 
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